Working with Git and Gerrit » History » Version 21

Karsten Dambekalns, 2010-08-12 13:32
Added a TOC

1 21 Karsten Dambekalns
{{>toc}}
2
3 1 Karsten Dambekalns
h1. Working with Git and Gerrit Code Review
4
5 20 Karsten Dambekalns
*Note:* We have not yet switched to Git and Gerrit, but are currently in the process of doing so. Planned date for the switch to git for FLOW3 and TYPO3 Phoenix projects is August 23rd.
6 1 Karsten Dambekalns
7 20 Karsten Dambekalns
For FLOW3 and TYPO3 Phoenix we are using Git and the Gerrit Code Review system to channel commits for review.
8 1 Karsten Dambekalns
9 20 Karsten Dambekalns
Now here is what you need to know to contribute, more details on Git are available in the "Pro Git book":http://progit.org/book/, the "Gerrit documentation":https://review.typo3.org/Documentation/index.html and other resources.
10 1 Karsten Dambekalns
11
h2. Git
12
13
h3. Getting Git
14
15
Firstly, if your machine doesn't already have it installed, get a copy of the 'git' version control system. This is available for many platforms from their upstream package repositories or, failing that, can be downloaded in both source and binary form from http://git-scm.com/download
16
17
If you look for a GUI tool to use with git, you can use the built-in GUI by starting gitk from the command line. For Windows there is "TortoiseGit":http://code.google.com/p/tortoisegit/, on the Mac you can use "gitx":http://gitx.frim.nl/ and for all platforms (Mac, Linux, Windows) there is "SmartGit":http://www.syntevo.com/smartgit/index.html - more tools exist, if those don't fit your needs.
18
19
h3. First-time Git setup
20
21
You must tell git a little about yourself so it knows who you are when doing commits. So simply do this:
22
23
<pre>$ git config --global user.name "John Doe"
24
$ git config --global user.email johndoe@example.com</pre>
25
26
More details and helpful configuration options are explained in the "Pro Git book, chapter 1.5":http://progit.org/book/ch1-5.html
27
28
h3. Getting the codebase
29
30 15 Karsten Dambekalns
*Note:* To clone what we call "distribution" or "application" a single clone command does not suffice! We used svn:externals, and a replacement for this is in git are "submodules". See below for a recipe to clone those types of project.
31 7 Karsten Dambekalns
32 12 Karsten Dambekalns
You can download the repository for a project by running one of
33
<pre>git clone git://git.typo3.org/<projectname>.git
34
git clone git://git.typo3.org/<projectname>.git <name-of-directory></pre>
35 13 Karsten Dambekalns
to place the clone in a directory called like the project or to place your clone in a specific directory.
36 19 Karsten Dambekalns
37
This will give you a complete copy of the repository and unless you are exceptionally short of either disk space, or time, is the approach we recommend. Unlike with Subversion, you are not just checking out a particular branch, but making a local copy of the project's whole revision history, across all of it's branches. This does make the download pretty large - but makes daily work with Git blazingly fast and allows for all the goodies of a distributed SCM.
38
39 1 Karsten Dambekalns
h3. Updating the local copy
40
41 15 Karsten Dambekalns
*Note:* To update what we call "distribution" or "application" a single pull command does not suffice! We used svn:externals, and a replacement for this is in git are "submodules". See below for a recipe to update those types of project.
42 7 Karsten Dambekalns
43 1 Karsten Dambekalns
When you want to update the local repository with the central one, running
44
<pre>git pull</pre>
45
will pull all of the new changes into your local repository, and merge those changes into your correct working tree. Note that whilst this is fine when you are browsing the repository, you may want to exercise more control over how upstream changes are merged into your development code.
46
47 7 Karsten Dambekalns
h3. Handling "superprojects" bundling multiple packages
48
49 15 Karsten Dambekalns
We used svn:externals a lot to combine some common configuration and code with assorted packages to form "distributions" (e.g. FLOW3 and TYPO3 Phoenix) or "applications" (e.g. the Blog). Git does have a replacement for svn:externals called submodules. It does not allow pointing to trunk instead of a fixed revision, so some care needs to be taken.
50 1 Karsten Dambekalns
51 15 Karsten Dambekalns
h4. Cloning a superproject
52 1 Karsten Dambekalns
53 15 Karsten Dambekalns
As usual you need to clone the distribution or application. Then do the following from the root folder of the new clone:
54
<pre>git submodule init
55
git submodule update</pre>
56 7 Karsten Dambekalns
57 16 Karsten Dambekalns
If you are lazy, the same effect can be achieved cloning like this right from the start:
58
<pre>git clone --recursive git://git.typo3.org/<projectname>.git</pre>
59
60 15 Karsten Dambekalns
*Important:* You must then install the gerrit commit-msg hook into the superproject as well as any new clone. This can be done with the help of @git submodule foreach@ as shown above.
61 7 Karsten Dambekalns
62 15 Karsten Dambekalns
h4. Updating a superproject
63 1 Karsten Dambekalns
64 17 Bastian Waidelich
When you update the superproject using @git pull@ the submodules will not be updated, you need to run an additional @git submodule update@ afterwards.
65 1 Karsten Dambekalns
66 15 Karsten Dambekalns
*Note:* Make sure you know what you are doing!
67 9 Karsten Dambekalns
68 15 Karsten Dambekalns
If you have pulled changes into a submodule or worked on the code there, submodule update will override the changes and check out the commit recorded in the superproject.
69 7 Karsten Dambekalns
70 15 Karsten Dambekalns
If you worked on a topic branch (very recommended!) you can simply checkout that branch again to get your work back.
71 7 Karsten Dambekalns
72 15 Karsten Dambekalns
If you want to update submodules prior to them being "officially" included in the superproject you can do that by
73
<pre>git checkout master
74
git pull</pre>
75
in the submodule root folder. So, to simulate what we had earlier with svn:externals pointing to trunk of all packages included with a superproject you can do this:
76
<pre>git submodule foreach 'git checkout master; git pull'</pre>
77 7 Karsten Dambekalns
78 2 Karsten Dambekalns
h3. Working on the code
79 1 Karsten Dambekalns
80 2 Karsten Dambekalns
We strongly recommend that you do all of your development upon 'topic branches.' This allows you to isolate multiple unrelated changes, and makes it easier to keep your tree in sync with the upstream one.
81 1 Karsten Dambekalns
82
Before creating a new topic branch, running
83
<pre>git fetch</pre>
84 2 Karsten Dambekalns
will make sure that your repository knows about the latest upstream changes (unlike @git pull@, this will not update any files that you may have checked out).
85 1 Karsten Dambekalns
86
To create a new topic branch:
87
<pre>git checkout -b <branch></pre>
88 2 Karsten Dambekalns
For example, to work on a patch to fix ereg warnings, based on the current development code, I would do:
89
<pre>git checkout -b fix-ereg-warnings origin/master</pre>
90 1 Karsten Dambekalns
This puts me on a new branch, ready to start writing code. All new development should be based upon the origin/master branch, submissions based upon other branches are unlikely to be accepted, unless they address issues that are solely present in that branch.
91
92 2 Karsten Dambekalns
To tell git about any new files you create as part of your patch
93
<pre>git add</pre>
94
is used. If your patch results in any new byproducts (cache, metadata files, etc) that git should not be tracking, please make sure that they're caught by the .gitignore mechanism. You can do this by checking that they don't appear in the output from @git status@
95 1 Karsten Dambekalns
96 2 Karsten Dambekalns
@git mv@ and @git rm@ are used to move and delete files respectively.
97 1 Karsten Dambekalns
98 2 Karsten Dambekalns
<pre>git commit -a</pre> is used to commit code to all of the files that git is currently tracking (that is, all of the files that you have checked out from the repository, and all those which you have run git add on)
99
100 1 Karsten Dambekalns
Be sure to explore the power of staging your changes: unlike with Subversion you can stage some changes to a file for commit while not (yet) stage other changes to the same file. This can come in very handy. To shelve work in progress, have a look at @git stash@.
101
102
h3. When you can't see the wood for the trees
103
104
If, in the middle of development, you discover that you've gone down a blind alley, and wish to go back to the state of your last commit
105
<pre>git reset --hard</pre>
106
will discard all of the changes you have made since the last commit, or
107
<pre>git checkout -f <file></pre>
108
will restore <file> to the state it was in at the last commit.
109
110
h3. Keeping up with upstream changes
111
112
If you're working on a long running development project, you will find that the point your created your topic branch rapidly recedes into history. At some point (and at least before you share your code with us), you'll probably want to update your tree. There are a number of ways of doing this.
113
114
If you haven't shared your tree with anyone else, then you can use
115 2 Karsten Dambekalns
<pre>git rebase <branch> <topic></pre>
116 1 Karsten Dambekalns
(Where <branch> is the name of the upstream branch - for example origin/master, and <topic> is the name of the topic branch you are currently working on)
117 2 Karsten Dambekalns
118
Note that git rebase changes your local history (it moves the branch point of your topic branch to the tip of the upstream branch), and is a bad idea if others have cloned your repository. See @man git-rebase@ for more details.
119 1 Karsten Dambekalns
120
If you can't rebase, then consider either merging the changes onto your local branch, or creating a new topic branch and cherry picking your changes onto it. The man pages for @git merge@ and @git cherry-pick@ provide more detail on these options.
121 2 Karsten Dambekalns
122 1 Karsten Dambekalns
h2. Sharing your code with us
123
124
How you work from this point onwards depends on how you intend making your code available to us. We're still happy to receive submission by patch (attach the patch to the issue you created or worked on using the Forge), but it makes it much easier for us if you push directly from your git tree to our code review system, gerrit.
125
126
h3. Registering with gerrit
127
128
To register with gerrit, visit https://review.typo3.org/ and log in using your typo3.org username. Click on 'Settings' -> 'Profile', and check that gerrit correctly knows your name and email address.
129
130
In order to be able to upload code, you now need to create a ssh key that gerrit can use to identify you, or tell gerrit about one that already exists.
131
132
To create a new ssh key, if you don't already have one, run
133
<pre>ssh-keygen -t rsa -f ~/.ssh/id_rsa</pre>
134
The public key for this is now stored in ~/.ssh/id_rsa.pub.
135
136
To tell gerrit about your key, log in, and go to 'Settings'. Select 'SSH Keys', and paste your public key into the "Add SSH Public Key" box, or click on the 'Open Key...' option to load it from the filesystem. Click on 'Add' to add the new public key.
137
To make things easier, set up OpenSSH so that it knows about the defaults for the gerrit server. Edit ~/.ssh/config, and add a section like:
138
<pre>Host review.typo3.org 
139
User <username> 
140
IdentityFile ~/.ssh/id_rsa 
141
Port 29418</pre>
142
(where username is what you logged in with, i.e. your typo3.org username)
143 18 Karsten Dambekalns
144 20 Karsten Dambekalns
h3. The change id hook
145
146
Gerrit introduces the concept of "Change IDs". This is a unique reference for a particular change, which remains constant regardless of any changes that are made to the implementation. This allows a single reference to be attached to a given modification, irrespective of any rewrites that may occur as a result of review comments. Manually maintaining change Ids is a pain, so gerrit provides a git hook which can be used to automatically add a change Id to any new modifications you create.
147
148
Because hooks cannot be transferred with a clone of a project, you must copy them to your .git/hooks directory and make them executable. The hook can be downloaded from the gerrit server by running the following in the top level of your git tree
149
<pre>scp -p -P 29418 review.typo3.org:hooks/commit-msg .git/hooks/</pre>
150
151
*Note:* Do this for every working copy you clone! Make sure the hook is marked as executable after it is installed (scp should take care of this)!
152
153
When using a superproject with submodules (see below), you can use this to install the hook for every submodule:
154
<pre>git submodule foreach 'scp -p -P 29418 review.typo3.org:hooks/commit-msg .git/hooks/'</pre>
155
156
If you cannot use scp (you should, it takes care of permissions automatically!), you can download the hook via HTTP, e.g. using @curl@, @wget@ or a browser, from
157
<pre>https://review.typo3.org/tools/hooks/commit-msg</pre>
158
159 1 Karsten Dambekalns
h3. Uploading to gerrit
160 18 Karsten Dambekalns
161 2 Karsten Dambekalns
When submitting to gerrit, it's important to realise that each commit in your branch will become a changeset in the upstream codebase, typically with no modification at our end. It is therefore important that these commits follow some simple rules...
162
163 1 Karsten Dambekalns
First, each commit should be complete. The rule "one change per commit, one commit per change" applies here. Each commit should build and test in its own right. Typically, this means that a change will be in a small number of commits. If, during development, you have created many more than this (for example, you've created a large number of bug fix commits), please use 'git rebase', or cherry pick these commits into a separate tree before uploading them.
164 2 Karsten Dambekalns
165 20 Karsten Dambekalns
Secondly, each commit should have a meaningful revision log message. The internals of git means that we can't easily edit these before pushing them into the tree, so we'd like you to get them right for us! A commit message should be comprised of a single 'subject' line (which must not end with a full stop), followed by a blank line, followed by one or more paragraphs explaining the purpose of the patch. Here is "more information on git revision log conventions":http://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html
166 2 Karsten Dambekalns
167 20 Karsten Dambekalns
The usual rules for relating to Forge issues and marking changes (task, bugfix, ...) apply (see FLOW3 CGL). To be able to search for issue numbers in gerrit, please add a "footer" line in the following format for every issue your commit relates to (in addition to any "fixes", "resolves" or "relates to" references):
168
<pre>Issue: #1234</pre>
169
170 2 Karsten Dambekalns
Once you have commits in this form, use
171 1 Karsten Dambekalns
<pre>git log -p origin/<branch>..HEAD</pre>
172 11 Karsten Dambekalns
(where <branch> is the upstream branch upon which you are basing this patch) to check that what you're giving us makes sense.
173
174 1 Karsten Dambekalns
Make sure the Change-Id line is in the commit message(s), if it is not, check the commit-msg hook and amend the commit to get the Change-Id in.
175 11 Karsten Dambekalns
176
Then, upload them to gerrit using
177 10 Karsten Dambekalns
<pre>git push ssh://review.typo3.org/<projectname> HEAD:refs/for/<branch></pre>
178 2 Karsten Dambekalns
(again <branch> is the name of the upstream branch that you are pushing the changes into, not the name of any local branch you may have been developing on).
179
180
In this case, 'refs/for' is a literal string. So, if you had been developing against master, you can upload your changes with:
181 10 Karsten Dambekalns
<pre>git push ssh://review.typo3.org/<projectname> HEAD:refs/for/master</pre>
182 2 Karsten Dambekalns
183 20 Karsten Dambekalns
This relies upon the ssh configuration you performed earlier. If it fails to work, please consult the "gerrit troubleshooting notes":https://review.typo3.org/Documentation/user-upload.html
184 2 Karsten Dambekalns
185
Assuming all has gone well, this will have added the entry to the code review queue. The output from git review will give you a change number - this is a unique reference for this particular set of changes. During review you'll be emailed with any comments anyone makes, and can respond to those comments using the gerrit web interface (see the section on reviewing, below). It's possible that issues with your change may be noticed during the review process, and you may be asked to revise it, or update changes to the tip of the tree.
186
187 1 Karsten Dambekalns
h3. Revising your change
188
189 2 Karsten Dambekalns
It's possible that your modifications won't be accepted first time. In this case, you need to revise your changes, and resubmit them to gerrit. Please note that this should always be done by modifying your original changeset, not by submitting a new change that makes the required fixes. Either @git commit --amend@, or @git rebase@ should be used to combine your changes with the original changeset, and then you should push this to gerrit with
190 7 Karsten Dambekalns
<pre>git push ssh://review.typo3.org/<projectname>.git <hash>:refs/changes/<number></pre>
191 2 Karsten Dambekalns
(where <hash> is the sha1 hash of the revised change, and <number> is the change number you received when you originally submitted the patch).
192
193
You can obtain the sha1 hash of a commit by using 'git show' (if it is on the tip of your current branch), or 'git log' (if it is in your history).
194
195 20 Karsten Dambekalns
Other mechanisms of listing the change to push are available - see the  "documentation":https://review.typo3.org/Documentation/user-upload.html for full details.
196 1 Karsten Dambekalns
197
h3. Updating your change
198 2 Karsten Dambekalns
199
It's possible that your change may have been made against a tree which is too old for it to apply to the tip. In this case, gerrit will let you know that there is a collision, and request that you update the change to the tip.
200
201
You can do this with
202
<pre>git rebase origin/master <topic></pre>
203
(assuming your patch is against the 'master' git branch, and lives on the <topic> branch).
204
205
And then simply resubmit your change in the same way as if you had been asked to revise it (see notes above)
206 1 Karsten Dambekalns
207
h3. Submitting by patch
208 2 Karsten Dambekalns
209
If all of this seems too daunting (and please don't let it put you off) you can still, of course, submit patches by email.
210
<pre>git diff HEAD</pre>
211
will give you the set of changes if you don't do local commits. If you make topic branches, and commit things locally, but don't want to go through the whole gerrit process,
212
<pre>git diff master..<topic></pre>
213
will give all of the changes between the branch point of you topic branch (assuming you branched from 'master') and the last commit.
214
You can attach those to issues in the Forge issue trackers as before. Note, however, by doing this you're making someone else take the patch, create a topic branch in their local tree, apply the patch, and push it into gerrit.
215
216
*Things would be much more efficient if you pushed into gerrit yourself. Please?*
217 4 Karsten Dambekalns
218 1 Karsten Dambekalns
h4. Committing Patches
219 4 Karsten Dambekalns
220
To commit a patch, make a new topic branch, and apply the patch to that branch. When committing, use
221
<pre>git commit --author="A.N. Author <an.author@example.org>"</pre>
222
to acknowledge the patch author (your name will still be there as the committer of the patch). Push the patch into gerrit as usual, and wait for reviews ...
223
224 1 Karsten Dambekalns
h2. Reviewing changes
225 2 Karsten Dambekalns
226 7 Karsten Dambekalns
We'll now look at how changes that have made it into gerrit can be reviewed. All code review now happens via the http://review.typo3.org interface. You should log in there as detailed above (using your typo3.org user credentials), and make sure that the email address points to somewhere you'll read regularly.
227 2 Karsten Dambekalns
228
You'll be presented with a list of patches requiring review or, if someone has asked, patches you've been explicitly requested to review. There are two types of review - Code Review and Verification. Code Review means that you have read through the code, and are satisified that it works properly, follows the tree's style, and generally doesn't suck. Verification means that you have taken a copy of the patch and tested it. We will automate the verification step through our CI server Hudson, but generally both should be perfomed (also) manually.
229 1 Karsten Dambekalns
230
To perform a code review, go through each of the diffs in the current changeset for the code you have decided to review. You can double click on a line to leave a comment. Once you have completed commenting, click on the 'Review' button that's about 3/4 of the way down the page containing the list of patch sets. You will then be asked to score the patch, with a range from -1 to +1. -1 means that you don't think the code should be applied, +1 means that it is good to apply. You can also leave further, general, comments for the patch submitter.
231 4 Karsten Dambekalns
232 1 Karsten Dambekalns
Note that no matter how many +1 or -1 comments a patch receives, the team members can override these to either permit or forbid submission by giving -2 or +2 on a patch ("gatekeeper" permissions). Also, at least one team member must approve a patch in this way before it can be submitted to the tree.
233 4 Karsten Dambekalns
234
A +2 means that the patch can be submitted. A -2 means that the patch is rejected. Note that these scores are not additive (2 +1s do not make a +2, 4 +1s don't cancel out a -2). This means that gatekeeper's scores are definitive - it doesn't matter what others may score, although that should probably be used as a guide.
235
236
In a specific project certain additional rules may applay as to who is allowed to give final acknowlegdement, so make sure to know what you are doing.
237 2 Karsten Dambekalns
238 1 Karsten Dambekalns
To verify the code, pull the git copy into your local tree, using the git command listed as part of the patch details. For sanity's sake, we'd strongly recommend you do this pull into a topic branch you've created for the purpose. Build it, test it, and report your results. Note that a single -1 to verification can block patch submission, so please use these options wisely. If in doubt, score 0 and leave your comments. Please indicate when verifying which platforms you have tested on, and what testing you performed.
239 4 Karsten Dambekalns
240
h2. Gatekeeper tasks
241
242
When a patch is reviewed and verified a few tasks are still open.
243
244
h3. Submitting the patch
245
246
Once a patch has both a +2 code review, and has verified OK, it may be submitted. A Submit button will appear on the patch's gerrit page - when clicked, the patch will be pushed into the git tree.
247
248
h3. Dealing with conflicts
249
250
You may find that attempting to cherry-pick a patch via gerrit results in a conflict. One way to address this is to modify the commit. Start with a current repository, cherry-pick the commit, fix it, and push back to gerrit.
251
252
<pre>git fetch
253
git checkout -b <new-topic-branch>
254
git cherry-pick <sha1 of contribution></pre>
255
The cherry-pick will give you a warning like
256 1 Karsten Dambekalns
<pre>Automatic cherry-pick failed.  After resolving the conflicts,
257 4 Karsten Dambekalns
mark the corrected paths with 'git add <paths>' or 'git rm <paths>' and commit the result.
258
When commiting, use the option '-c 822e735' to retain authorship and message.</pre>
259
if it fails. At this point, resolve the conflict. You can use @git diff@ to determine what it is. Then, continue:
260
<pre>git add <file with resolved conflict> [<next file with resolved conflict>]
261
git commit -c <commit hash from cherry pick></pre>
262
Confirm you will push the right thing:
263
<pre>git log -p origin/master..HEAD</pre>
264
Then, push the change back to the open issue in gerrit.
265 7 Karsten Dambekalns
<pre>git push ssh://review.typo3.org/openafs <hash>:refs/changes/<gerrit change number></pre>
266 4 Karsten Dambekalns
267
h3. Modifying a change in gerrit
268
269
The typical gerrit workflow expects that instead of gatekeepers modifying changes in gerrit, they will simply leave review comments and expect the committer to fix their patch to address those comments themselves (the theory being that this educates the committers, and thus reduces the load on the gatekeepers in the long term). However, if you want to modify someone's change, you can do so.
270
Simply fetch the change from gerrit as if you were going to verify it, making sure that you do that fetch into a topic branch. Then, make your modifications and run
271
<pre>git commit --amend --author "A.N. Author <a.n.author@example.org>"</pre>
272
Then push this commit into gerrit as a replacement from the original change (see above for details of exactly how to do this)
273
274 2 Karsten Dambekalns
h2. Thanks
275
276
... to the folks over at "OpenAFS":http://openafs.org/ that made us stumble over gerrit and whose wiki page we shamelessly used as a base for this page.
277
278 1 Karsten Dambekalns
... to Sebastian Kurf├╝rst, Peter Niederlag for their help with git, gerrit and redmine.
279 2 Karsten Dambekalns
280 5 Karsten Dambekalns
... to the TYPO3 Core Team for being open towards a change in SCM flavor.