1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
|
Content-type: text/asciidoc
Abstract: When a vulnerability is reported, we follow these guidelines to
assess the vulnerability, create and review a fix, and coordinate embargoed
security releases.
How we coordinate embargoed releases
------------------------------------
To protect Git users from critical vulnerabilities, we do not just release
fixed versions like regular maintenance releases. Instead, we coordinate
releases with packagers, keeping the fixes under an embargo until the release
date. That way, users will have a chance to upgrade on that date, no matter
what Operating System or distribution they run.
The `git-security` mailing list
-------------------------------
Responsible disclosures of vulnerabilities, analysis, proposed fixes as
well as the orchestration of coordinated embargoed releases all happen on the
`git-security` mailing list at <git-security@googlegroups.com>.
In this context, the term "embargo" refers to the time period that information
about a vulnerability is kept under wraps and only shared on a need-to-know
basis. This is necessary to protect Git's users from bad actors who would
otherwise be made aware of attack vectors that could be exploited. "Lifting the
embargo" refers to publishing the version that fixes the vulnerabilities.
Audience of the `git-security` mailing list
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Anybody may contact the `git-security` mailing list by sending an email
to <git-security@googlegroups.com>, though the archive is closed to the
public and only accessible to subscribed members.
There are a few dozen subscribed members: core Git developers who are trusted
with addressing vulnerabilities, and stakeholders (i.e. owners of products
affected by security vulnerabilities in Git).
Most of the discussions revolve around assessing the severity of the reported
issue (including the decision whether the report is security-relevant or can be
redirected to the public mailing list), how to remediate the issue, determining
the timeline of the disclosure as well as aligning priorities and
requirements.
Communications
~~~~~~~~~~~~~~
If you are a stakeholder, it is a good idea to pay close attention to the
discussions, as pertinent information may be buried in the middle of a lively
conversation that might not look relevant to your interests. For example, the
tentative timeline might be agreed upon in the middle of discussing code
comment formatting in one of the patches and whether or not to combine fixes
for multiple, separate vulnerabilities into the same embargoed release. Most
mail threads are not usually structured specifically to communicate
agreements, assessments or timelines.
Typical timeline
----------------
- A potential vulnerability is reported to the `git-security` mailing list.
- The members of the git-security list start a discussion to give an initial
assessment of the severity of the reported potential vulnerability.
We aspire to do so within a few days.
- After discussion, if consensus is reached that it is not critical enough
to warrant any embargo, the reporter is redirected to the public Git mailing
list. This ends the reporter's interaction with the `git-security` list.
- If it is deemed critical enough for an embargo, ideas are presented on how to
address the vulnerability.
- Usually around that time, the Git maintainer or their delegate(s) open a draft
security advisory in the `git/git` repository on GitHub (see below for more
details).
- Code review can take place in a variety of different locations,
depending on context. These are: patches sent inline on the git-security list,
a private fork on GitHub associated with the draft security advisory, or the
git/cabal repository.
- Contributors working on a fix should consider beginning by sending
patches to the git-security list (inline with the original thread), since they
are accessible to all subscribers, along with the original reporter.
- Once the review has settled and everyone involved in the review agrees that
the patches are nearing the finish line, the Git maintainer, and others
determine a release date as well as the release trains that are serviced. The
decision regarding which versions need a backported fix is based on input from
the reporter, the contributor who worked on the patches, and from
stakeholders. Operators of hosting sites who may want to analyze whether the
given issue is exploited via any of the repositories they host, and binary
packagers who want to make sure their product gets patched adequately against
the vulnerability, for example, may want to give their input at this stage.
- While the Git community does its best to accommodate the specific timeline
requests of the various binary packagers, the nature of the issue may preclude
a prolonged release schedule. For fixes deemed urgent, it may be in the best
interest of the Git users community to shorten the disclosure and release
timeline, and packagers may need to adapt accordingly.
- Subsequently, branches with the fixes are pushed to the git/cabal repository.
- The tags are created by the Git maintainer and pushed to the same repository.
- The Git for Windows, Git for macOS, BSD, Debian, etc. maintainers prepare the
corresponding release artifacts, based on the tags created that have been
prepared by the Git maintainer.
- The release artifacts prepared by various binary packagers can be
made available to stakeholders under embargo via a mail to the
`git-security` list.
- Less than a week before the release, a mail with the relevant information is
sent to <distros@vs.openwall.org> (see below), a list used to pre-announce
embargoed releases of open source projects to the stakeholders of all major
distributions of Linux as well as other OSes.
- Public communication is then prepared in advance of the release date. This
includes blog posts and mails to the Git and Git for Windows mailing lists.
- On the day of the release, at around 10am Pacific Time, the Git maintainer
pushes the tag and the `master` branch to the public repository, then sends
out an announcement mail.
- Once the tag is pushed, the Git for Windows maintainer publishes the
corresponding tag and creates a GitHub Release with the associated release
artifacts (Git for Windows installer, Portable Git, MinGit, etc).
- Git for Windows release is then announced via a mail to the public Git and
Git for Windows mailing lists as well as via a tweet.
- Ditto for distribution packagers for Linux and other platforms:
their releases are announced via their preferred channels.
- A mail to <oss-security@lists.openwall.org> (see below for details) is sent
as a follow-up to the <distros@vs.openwall.org> one, describing the
vulnerability in detail, often including a proof of concept of an exploit.
Note: The Git project makes no guarantees about timelines, but aims to keep
embargoes reasonably short in the interest of keeping Git's users safe.
Opening a Security Advisory draft
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The first step is to https://github.com/git/git/security/advisories/new[open
an advisory]. Technically, this is not necessary. However, it is the most
convenient way to obtain the CVE number and it give us a private repository
associated with it that can be used to collaborate on a fix.
Notifying the Linux distributions
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
At most two weeks before release date, we need to send a notification to
<distros@vs.openwall.org>, preferably less than 7 days before the release date.
This will reach most (all?) Linux distributions. See an example below, and the
guidelines for this mailing list at
https://oss-security.openwall.org/wiki/mailing-lists/distros#how-to-use-the-lists[here].
Once the version has been published, we send a note about that to oss-security.
As an example, see https://www.openwall.com/lists/oss-security/2019/12/13/1[the
v2.24.1 mail];
https://oss-security.openwall.org/wiki/mailing-lists/oss-security[Here] are
their guidelines.
The mail to oss-security should also describe the exploit, and give credit to
the reporter(s): security researchers still receive too little respect for the
invaluable service they provide, and public credit goes a long way to keep them
paid by their respective organizations.
Technically, describing any exploit can be delayed up to 7 days, but we usually
refrain from doing that, including it right away.
As a courtesy we typically attach a Git bundle (as `.tar.xz` because the list
will drop `.bundle` attachments) in the mail to distros@ so that the involved
parties can take care of integrating/backporting them. This bundle is typically
created using a command like this:
git bundle create cve-xxx.bundle ^origin/master vA.B.C vD.E.F
tar cJvf cve-xxx.bundle.tar.xz cve-xxx.bundle
Example mail to distros@vs.openwall.org
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
....
To: distros@vs.openwall.org
Cc: git-security@googlegroups.com, <other people involved in the report/fix>
Subject: [vs] Upcoming Git security fix release
Team,
The Git project will release new versions on <date> at 10am Pacific Time or
soon thereafter. I have attached a Git bundle (embedded in a `.tar.xz` to avoid
it being dropped) which you can fetch into a clone of
https://github.com/git/git via `git fetch --tags /path/to/cve-xxx.bundle`,
containing the tags for versions <versions>.
You can verify with `git tag -v <tag>` that the versions were signed by
the Git maintainer, using the same GPG key as e.g. v2.24.0.
Please use these tags to prepare `git` packages for your various
distributions, using the appropriate tagged versions. The added test cases
help verify the correctness.
The addressed issues are:
<list of CVEs with a short description, typically copy/pasted from Git's
release notes, usually demo exploit(s), too>
Credit for finding the vulnerability goes to <reporter>, credit for fixing
it goes to <developer>.
Thanks,
<name>
....
Example mail to oss-security@lists.openwall.com
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
....
To: oss-security@lists.openwall.com
Cc: git-security@googlegroups.com, <other people involved in the report/fix>
Subject: git: <copy from security advisory>
Team,
The Git project released new versions on <date>, addressing <CVE>.
All supported platforms are affected in one way or another, and all Git
versions all the way back to <version> are affected. The fixed versions are:
<versions>.
Link to the announcement: <link to lore.kernel.org/git>
We highly recommend to upgrade.
The addressed issues are:
* <list of CVEs and their explanations, along with demo exploits>
Credit for finding the vulnerability goes to <reporter>, credit for fixing
it goes to <developer>.
Thanks,
<name>
....
|
