Effective programming practices for economists. 2. Version Control

Transcription

1 Effective programming practices for economists 2. Version Control Hans-Martin von Gaudecker Department of Economics, Universität Mannheim Licensed under the Creative Commons Attribution License 1/99

2 Problem 1: Collaboration Collaboration is becoming increasingly important: Science: [... ] half of EU research articles had international co-authors in 2007, more than twice the level of two decades ago (Hand, 2010) Economics: In the 1970s only 30 percent of the articles in the top five journals were coauthored. In the 1990s about 60 percent were coauthored. In the longer run the trend is even more striking: in 1959 only 3 percent of the articles in the JPE were coauthored. (Ellison, 2002) Need to have effective means for working with others. You are somebody else in the future as well... Same issue is to keep multiple machines in sync. Licensed under the Creative Commons Attribution License 2/99

3 Problem 2: Going back in time How many times in the process does one believe a paper is finished? And when you think it is done, enter... co-authors, supervisor, conference participants, referees. Often long lags in between changes in beliefs. Project changes in the meantime. Suddenly you re not able to reproduce the results you had at the time you submitted {paper, thesis}. Need to keep track of older versions. Licensed under the Creative Commons Attribution License 3/99

4 Ad-hoc solutions Sending s? Might work for a little document but entire directory structures? Even then very difficult to search for specific versions. Back up stuff regularly: my_project my_project_v0 my_project_v1 my_project_v2 You re certain to miss the important version. Better things to waste your energy on research than on such mindless stuff. Recurring message: Computers are much better at solving repetitive tasks than humans are. So let the machines do the work. Licensed under the Creative Commons Attribution License 4/99

5 Ad-hoc solutions You Your coauthor (Your future self) Licensed under the Creative Commons Attribution License 5/99

6 Ad-hoc solutions You Your coauthor (Your future self) Licensed under the Creative Commons Attribution License 6/99

7 The right solution You Your coauthor (Your future self) Licensed under the Creative Commons Attribution License 7/99

8 The right solution Repository with all versions of files. You Your coauthor (Your future self) Licensed under the Creative Commons Attribution License 8/99

9 The right solution Repository with all versions of files. You Working copies, each with one version of files Your coauthor (Your future self) Licensed under the Creative Commons Attribution License 9/99

10 The right solution Repository with all versions of files. check in / commit You Working copies, each with one version of files Your coauthor (Your future self) Licensed under the Creative Commons Attribution License 10/99

11 The right solution Repository with all versions of files. check in / commit check out (initialise) You Working copies, each with one version of files Your coauthor (Your future self) Licensed under the Creative Commons Attribution License 11/99

12 The right solution Repository with all versions of files. check in / commit check out (initialise) / update You Working copies, each with one version of files Your coauthor (Your future self) Licensed under the Creative Commons Attribution License 12/99

13 Checking out a populated repository Right-click in the area of the PyDev Package Explorer, select New... Project... Licensed under the Creative Commons Attribution License 13/99

14 Checking out a populated repository Expand the SVN category, select Project from SVN and click Next. Licensed under the Creative Commons Attribution License 14/99

15 Checking out a populated repository Licensed under the Creative Commons Attribution License 15/99

16 Checking out a populated repository Fill in the information as needed, here: Don t forget the s of Add svn/project-identifier to the address of the collaboration server You can find out the project identifier in Settings/Information on the project s web page. For your group X, it should be prog-econ-2010-grp-x Click Next. On the following screen, click on Head Revision and Finish. Licensed under the Creative Commons Attribution License 16/99

17 Checking out a populated repository Select Check out as a project... and click on Finish. You may change the name of the project if you do not like the default. Eclipse will create a directory with that name in your workspace. Licensed under the Creative Commons Attribution License 17/99

18 How version numbering works Every time you commit a changed file to the repository, this file and all directories higher up in the hierarchy get a new version number. Means that a revision number always uniquely identifies the entire repository. But you can quickly see when something (or something further down the hierarchy) changed the last time. On the collaboration server (the repository ), all versions are kept. In your working copy, you only see one. Generally the latest version, but easy to check out previous ones as well. Licensed under the Creative Commons Attribution License 18/99

19 The basic cycle: Check what has changed Right-click on the directory you want to check... Licensed under the Creative Commons Attribution License 19/99

20 The basic cycle: Check what has changed One file changed, see entire history at the bottom of the page. Licensed under the Creative Commons Attribution License 20/99

21 The basic cycle: Check what has changed After double-click on changed file. Licensed under the Creative Commons Attribution License 21/99

22 The basic cycle: Check what has changed Double-clicking a tab maximises it (and back). Licensed under the Creative Commons Attribution License 22/99

23 The basic cycle: Update your working copy Aside: Turn the SVN console on to see what s happening. Licensed under the Creative Commons Attribution License 23/99

24 The basic cycle: Update your working copy Right-click on the folder to update, and... Licensed under the Creative Commons Attribution License 24/99

25 The basic cycle: Update your working copy exercise solution template.tex was modified. Licensed under the Creative Commons Attribution License 25/99

26 The basic cycle: Changing your working copy You still need to do the real work... Modify files. Add files and directories. Here: Put a file named paper template.tex in the templates directory. Do that outside Eclipse. Need to hit F5 / right-click and Refresh on a directory somewhere up the hierarchy from the added file. Here: either templates or prog-econ-comm Licensed under the Creative Commons Attribution License 26/99

27 The basic cycle: Changing your working copy Meaning of the symbols:? The file has not been added to SVN yet it is unversioned. > The directory has changed. Licensed under the Creative Commons Attribution License 27/99

28 The basic cycle: Committing your changes Add the new file to version control. Licensed under the Creative Commons Attribution License 28/99

29 The basic cycle: Committing your changes Note how symbol has changed, but there is no version number. Licensed under the Creative Commons Attribution License 29/99

30 The basic cycle: Committing your changes Pick a meaningful comment. Licensed under the Creative Commons Attribution License 30/99

31 The basic cycle: Committing your changes Done! The two directories up the hierarchy will get new version numbers in the next update they are included in. Eclipse/Subversive will automatically include files we put in the directories, so manually adding them was not necessary (but it is required in other clients!). Why does the root directory appear to have changed? Licensed under the Creative Commons Attribution License 31/99

32 Eclipse does not show you everything... Check by attempting a commit from the root directory: Licensed under the Creative Commons Attribution License 32/99

33 Keep only source code under version control.project contains internal Eclipse information. Only relevant for your local machine. Changes far too often, distracts from real changes. In general - what to put under VC? An early version of these slides pdf file contained: /Author()/Title()/Subject()/Creator(LaTeX with beamer [...] /CreationDate (D: ) /ModDate (D: ) Will change every time you run pdflatex, even if there are no substantive changes whatsoever and thus lead to many fake changes of the repository. So keep only sources under VC. Original data and source code from statistics programs, L A TEX, etc.. Exception: Use it to keep project in sync across computers and there is a step you can only run on one machine (e.g. on a cluster). Licensed under the Creative Commons Attribution License 33/99

34 Setting properties on directories svn:ignore Right-click on the directory in question... Licensed under the Creative Commons Attribution License 34/99

35 Setting properties on directories svn:ignore Each property has a name... Usually pre-defined, but could be user-defined. We will meet: svn:ignore svn:externals svn:keywords... and a value Depending on the property, the value requires a certain format. Right-click in the area to add a property. Licensed under the Creative Commons Attribution License 35/99

36 Setting properties on directories svn:ignore Licensed under the Creative Commons Attribution License 36/99

37 Setting properties on directories svn:ignore Licensed under the Creative Commons Attribution License 37/99

38 Setting properties on directories svn:ignore Every line specifies a pattern to ignore. Could specify entire file/directory names, or patterns means match any number of characters of any type (this includes no character at all). The.svn directories are automatically ignored (of course ). Can always override the svn:ignore property by manually adding a file as we did before. E.g. for pdf s that are sources (data documentation, references,... ). Here we want to apply the properties to all directories. (Note for reference that you won t understand now) Be careful with the recursive option if you have directories via the svn:externals property further down the hierarchy, it will be applied to those as well. Licensed under the Creative Commons Attribution License 38/99

39 Setting properties on directories svn:ignore The content has not changed, but the properties have. Now the.project file is ignored indeed. Licensed under the Creative Commons Attribution License 39/99

40 Trying to commit from an outdated working copy Ooops. You always have to commit from the latest version. The root directory and templates directory are still at revision 3. Run an svn update from the root and everything works as expected. Licensed under the Creative Commons Attribution License 40/99

41 Why is that so? While you are working on the project, someone else might be working on it as well. Unless you lock a resource (file, directory) nobody else except you will be able to commit changes to this resource. Almost never necessary, far too restrictive. What if changes get into the way of each other? Merge your version and the latest one from the repository. For safety, Subversion requires you to do that in the working copy, i.e. during an update and not during a commit. Two types of changes: Non-conflicting: Subversion will automatically merge them (but you better check). Conflicting: You have to edit conflicts yourself (with some help from specialised editors). Licensed under the Creative Commons Attribution License 41/99

42 Add another file type to ignore to template... Licensed under the Creative Commons Attribution License 42/99

43 ... and get the commit failed message again... Licensed under the Creative Commons Attribution License 43/99

44 ... but why? Licensed under the Creative Commons Attribution License 44/99

45 A couple of lines were added, no problem... Licensed under the Creative Commons Attribution License 45/99

46 Handling non-conflicting changes: Just update Licensed under the Creative Commons Attribution License 46/99

47 Handling non-conflicting changes The file with my local modifications and the one from the repository with Ina s changes were merged by Subversion. Nothing left to do, just commit version 7, see below. But always good to check things the way we did. That changes do not mean conflicts to Subversion s algorithm does not mean they are no conflicts conceptually. E.g. insert two functions with the same name at different locations in a file. Licensed under the Creative Commons Attribution License 47/99

48 Added a line and a typo in revision 8 Licensed under the Creative Commons Attribution License 48/99

49 How we end up at the next screen... Both me and Ina realise the typo in *.pdfsyc. Ina corrects it back to *.pdfsync, commits her change. The repository is now at revision 9. I change it to *.pdfs* and try to commit. Pretty stupid change, actually. Unless there is a clear reason for a wildcard, better be explicit. The commit attempt leads to the well-known error. Run an update without further checks. Licensed under the Creative Commons Attribution License 49/99

50 What conflicting changes look like Licensed under the Creative Commons Attribution License 50/99

51 How to resolve conflicts Licensed under the Creative Commons Attribution License 51/99

52 How to resolve conflicts Only the left pane is editable, of course. You could: Accept the change from the repository, leave your local change as is, change the left pane to something entirely new. We will opt for taking the change from the repository. Licensed under the Creative Commons Attribution License 52/99

53 Sometimes a three-way merge is useful See common base (here: revision 8 with typo) in top pane. Licensed under the Creative Commons Attribution License 53/99

54 Finish by marking everything as merged Licensed under the Creative Commons Attribution License 54/99

55 The final state of affairs Everything is at revision 9 and Subversion sees that there are no local changes anymore. Licensed under the Creative Commons Attribution License 55/99

56 Better alternative: Synchronise with repository When you get the previous commit failed message, go to... Licensed under the Creative Commons Attribution License 56/99

57 Better alternative: Synchronise with repository Licensed under the Creative Commons Attribution License 57/99

58 If you find yourself resolving conflicts all the time... it is a sign that something is wrong with the workflow in your project. Not talking to co-authors as often as you should? Responsibilities not clearly assigned? Subversion helps you detect this. The conflict resolution mechanisms we have seen are limited to text files. I.e., human readable. Most parts of pdf-files, MS Office files,.mat or.dta data formats,... are not. The deeper reason for having only sources under VC. Subversion will mark such files as binary. If you need to use Word, a workaround in case of a conflict is to download the latest version of the file from the repository (easy via Redmine) and use Word s built-in document comparison. Licensed under the Creative Commons Attribution License 58/99

59 The case for plain L A TEX So use simple text files for writing papers. L A TEX is the de-facto standard. Will see later that it is also very useful for getting closer to that red button. What about the middle ground Scientific Workplace, LyX? Better, but not good. Also format the source code. Not designed for human readability merging conflicts is a pain in the. More control over what is going on. Learning curve is steep. But shortly, your mind will ignore the markup commands when reading L A TEX source code. Licensed under the Creative Commons Attribution License 59/99

60 Using Subversion to go back in time One of the big benefits you get from using a version control system. You can use it as an infinite undo-button. Its coarseness depends on how often you commit. Better too often than too seldomly. Try to commit chunks that share a logical connection. Distinguish between two cases: Local changes that you do not want to keep. Changes you committed already go back to a previous revision. Licensed under the Creative Commons Attribution License 60/99

61 Reverting local changes Licensed under the Creative Commons Attribution License 61/99

62 Reverting local changes Licensed under the Creative Commons Attribution License 62/99

63 Reverting local changes After this, a dialogue will pop up asking whether you really want to revert your changes. Click OK and you re done. Licensed under the Creative Commons Attribution License 63/99

64 Going back to a previous state of the repository Now we look at what to do when I committed the previous changes as revision 12, instead of reverting. Realise it was nonsense, want to go back to revision 11. Licensed under the Creative Commons Attribution License 64/99

65 Going back to a previous state of the repository Licensed under the Creative Commons Attribution License 65/99

67 Going back to a previous state of the repository After clicking OK, the Synchronise perspective will come up, but it is not very helpful here. So go back to PyDev perspective. Licensed under the Creative Commons Attribution License 67/99

69 Merging with prior versions is not ideal Licensed under the Creative Commons Attribution License 69/99

70 Merging with prior versions is not ideal... Scientific work always involves trying lots of things, some of them work, some do not Licensed under the Creative Commons Attribution License 70/99

71 The better solution: Branching Licensed under the Creative Commons Attribution License 71/99

72 The better solution: Branching Branch Experimental Trunk Licensed under the Creative Commons Attribution License 72/99

76 Taking a snapshot of the repository Tag As submitted Branch Experimental Trunk Licensed under the Creative Commons Attribution License 76/99

77 Recommended repository layout Trunk Branches Tags Licensed under the Creative Commons Attribution License 77/99

78 Recommended repository layout Trunk Branches Tags Eclipse can create this structure for you in a fresh repository. Licensed under the Creative Commons Attribution License 78/99

79 Creating a new project in Redmine When you create a project in Redmine, you will automatically get a repository (wait for a couple of minutes). Licensed under the Creative Commons Attribution License 79/99

80 Initialising a virgin repository Licensed under the Creative Commons Attribution License 80/99

87 Tagging in action Licensed under the Creative Commons Attribution License 87/99

90 A little more on Subversion s internals In each directory that Subversion knows about, your local working copy contains a hidden folder.svn. Stores all information about its parent direcory that Subversion needs to know. In particular, it contains information on: the location of the repository; files and directories in that folder that Subversion knows about; properties like svn:ignore; etc. Licensed under the Creative Commons Attribution License 90/99

91 A little more on Subversion s internals Consequence is that when moving entire directories, you have to think about the.svn directory. (Subversive for) Eclipse assumes you want to do to the repository what you did to your working copy (e.g. move a directory from one place to another). It issues the relevant commands implicitly, but this does not work always, e.g. moving nested directory structures does not seem to work smoothly. When you move things in the Windows Explorer, the information in the.svn directories does not change. Might lead to conflicting information. For TortoiseSVN, see: Licensed under the Creative Commons Attribution License 91/99

92 When everything stops working... Don t panic!!! You always have the situation from the last commit in the repository. So be sure to commit frequently. Always solve problems immediately so that you won t loose much information should you have to go back. Try out the solution strategies on the next slide in the order in which they appear. Licensed under the Creative Commons Attribution License 92/99

93 When everything stops working... First attempt: svn cleanup. If that doesn t work: In the Windows Explorer, move all files with important local changes to some safe place outside the working copy. svn revert, see whether you can update. If so, move back the files with changes and commit. If that doesn t work: In the Windows Explorer, delete the directory in question and run an svn update on its parent directory. Then move back the files with changes and commit. If directory that causes trouble is the one you checked out, just delete the entire directory and check it out anew. Licensed under the Creative Commons Attribution License 93/99

94 Moving a clean folder somewhere Sometimes you need a directory in a different repository, etc.. If you moved it naïvely in the Windows Explorer, the internal.svn directories would get into each others way. Elegant solution: svn export moves everything under Version Control to a new directory. Access it from context menus. If you re past that (i.e. you moved naïvely and now have the conflict)... Turn on view hidden files somewhere in your Explorer View menu. Now delete the.svn directories from the directories that you moved. Add them to the new repository as usual. Licensed under the Creative Commons Attribution License 94/99

95 A L A TEX primer Most important concept: You are responsible for the structure. L A TEX will do the design based on that structure. Concept also called markup. Basic structure of a L A TEX source file. Commands. References: Bibliographies. Use manager, e.g. Jabref on all platforms, BibDesk on Mac. Best book: Kopka and Daly (2004). Disclaimer. Very useful reference resources: Licensed under the Creative Commons Attribution License 95/99

96 Aside: Eclipse issues Upon first start, Eclipse will show you a welcome screen. Just click it away to get the usual perspective. Eclipse will ask you at some point whether you want to upload your usage data, which developers will use to improve its usability. Decide for yourself what you want to do, performance is unaffected. Licensed under the Creative Commons Attribution License 96/99

97 References I Ellison, Glenn (2002). The Slowdown of the Economics Publishing Process. In: The Journal of Political Economy 110.5, pp Hand, Eric (2010). Big Science Spurs Collaborative Trend. In: Nature News 463. Available at Kopka, Helmut and Patrick W. Daly (2004). Guide to L A TEX. 4 th. Addison Wesley / Pearson Education. Licensed under the Creative Commons Attribution License 97/99

98 Acknowledgements and revision number This course is designed after and borrows a lot from the Software Carpentry course designed by Greg Wilson for scientists and engineers. The Software Carpentry course material is made available under a Creative Commons Attribution License, as is this course s material. Last changed revision: 202 Last changed date: :49: (Mon, 01 Nov 2010) Licensed under the Creative Commons Attribution License 98/99

99 License for the course material [Links to the full legal text and the source text for this page.] You are free: to Share to copy, distribute and transmit the work to Remix to adapt the work Under the following conditions: Attribution You must attribute the work in the manner specified by the author or licensor (but not in any way that suggests that they endorse you or your use of the work). With the understanding that: Waiver Any of the above conditions can be waived if you get permission from the copyright holder. Public Domain Where the work or any of its elements is in the public domain under applicable law, that status is in no way affected by the license. Other Rights In no way are any of the following rights affected by the license: Your fair dealing or fair use rights, or other applicable copyright exceptions and limitations; The author s moral rights; Rights other persons may have either in the work itself or in how the work is used, such as publicity or privacy rights. Notice For any reuse or distribution, you must make clear to others the license terms of this work. The best way to do this is with a link to this web page.