Repository navigation
Keep Storage emulator object keys literal instead of path.join-normalizing them - #11307
SahinurDEV wants to merge 5 commits into
Conversation
…izing them The Storage emulator checked security rules against the requested object name but stored and looked up objects under path.join(bucket, object), which collapses '.' and '..' segments. A name like users/A/../B/photo.png was authorized as users/A/... and then read, overwrote or deleted users/B/photo.png. Cloud Storage treats object names literally, so key objects by bucket and literal name instead. Imports of old Windows exports, whose names use backslash separators, are still mapped to '/'. Fixes firebase#11301
There was a problem hiding this comment.
Code Review
This pull request fixes an issue in the Storage emulator where object names containing . or .. segments were stored under their normalized names, potentially leading to unauthorized reads, overwrites, or deletions of other objects. This is resolved by avoiding path.join (which collapses these segments) and instead using direct string interpolation for paths. Additionally, the import logic for older Windows-exported paths has been simplified. Feedback on the changes includes addressing a regex bug in the getPathSep helper function and ensuring that temporary directories created during the new export/import tests are properly cleaned up using rmSync in a try...finally block.
| let fileName = metadata.name; | ||
| const objectNameSep = getPathSep(fileName); | ||
| // Replace all file separators with that of current platform for compatibility | ||
| if (fileName !== path.sep) { | ||
| fileName = fileName.split(objectNameSep).join(path.sep); | ||
| if (getPathSep(fileName) === "\\") { | ||
| fileName = fileName.split("\\").join("/"); | ||
| } |
There was a problem hiding this comment.
The helper function getPathSep has a regex bug where the character class [\/|\\\\] includes a literal | character (which is treated as a literal character rather than a logical OR inside []). This means any filename containing a | character would have it incorrectly identified as a path separator. Since getPathSep is only used here, we can avoid using it entirely and match the first path separator correctly using a standard regex match.
| let fileName = metadata.name; | |
| const objectNameSep = getPathSep(fileName); | |
| // Replace all file separators with that of current platform for compatibility | |
| if (fileName !== path.sep) { | |
| fileName = fileName.split(objectNameSep).join(path.sep); | |
| if (getPathSep(fileName) === "\\") { | |
| fileName = fileName.split("\\").join("/"); | |
| } | |
| let fileName = metadata.name; | |
| const firstSep = fileName.match(/[\/\\]/)?.[0]; | |
| if (firstSep === "\\\\") { | |
| fileName = fileName.split("\\\\").join("/"); | |
| } |
There was a problem hiding this comment.
Good catch, thanks. getPathSep was existing code, but this PR relies on its result, so a name like a|b\c.txt from an old Windows export was seen as using | and its backslashes were not mapped to /. In 78daee2 I removed getPathSep (this was its only caller) and check the first separator directly with /[/\\]/.exec(fileName)?.[0] === "\\". I compared against a single \ rather than the suggested "\\\\": the match is always one character, so a two-backslash string would never equal it and the Windows mapping would stop working. I also added a test that imports a|b\c.txt and reads it back as a|b/c.txt.
| @@ -1,5 +1,7 @@ | |||
| import { expect } from "chai"; | |||
| import { mkdtempSync } from "fs"; | |||
There was a problem hiding this comment.
Thanks, done in 78daee2: the spec now imports rmSync and uses it to delete the temporary directories.
| it("should keep the objects apart through export and import", async () => { | ||
| const storageLayer = getStorageLayer(ALWAYS_TRUE_RULES_VALIDATOR); | ||
| await uploadDecoded(storageLayer, "bucket", "users/B/photo.png", "from B"); | ||
| await uploadDecoded(storageLayer, "bucket", "users/A/../B/photo.png", "from A"); | ||
| const exportDir = mkdtempSync(path.join(tmpdir(), "storage-export-")); | ||
| await storageLayer.export(exportDir, { initiatedBy: "test" }); | ||
|
|
||
| const imported = new StorageLayer( | ||
| "project", | ||
| new Map(), | ||
| new Map(), | ||
| ALWAYS_TRUE_RULES_VALIDATOR, | ||
| ALWAYS_TRUE_ADMIN_CREDENTIAL_VALIDATOR, | ||
| new Persistence(mkdtempSync(path.join(tmpdir(), "storage-import-"))), | ||
| new StorageCloudFunctions("project"), | ||
| ); | ||
| imported.import(exportDir, { initiatedBy: "test" }); | ||
|
|
||
| expect(await readData(imported, "users/B/photo.png")).to.equal("from B"); | ||
| expect(await readData(imported, "users/A/../B/photo.png")).to.equal("from A"); | ||
| }); |
There was a problem hiding this comment.
The temporary directories created with mkdtempSync are never cleaned up, which can clutter the system's temporary folder over time. Wrapping the test logic in a try...finally block ensures that these directories are always deleted, even if the test assertions fail.
it("should keep the objects apart through export and import", async () => {
const storageLayer = getStorageLayer(ALWAYS_TRUE_RULES_VALIDATOR);
await uploadDecoded(storageLayer, "bucket", "users/B/photo.png", "from B");
await uploadDecoded(storageLayer, "bucket", "users/A/../B/photo.png", "from A");
const exportDir = mkdtempSync(path.join(tmpdir(), "storage-export-"));
const importDir = mkdtempSync(path.join(tmpdir(), "storage-import-"));
try {
await storageLayer.export(exportDir, { initiatedBy: "test" });
const imported = new StorageLayer(
"project",
new Map(),
new Map(),
ALWAYS_TRUE_RULES_VALIDATOR,
ALWAYS_TRUE_ADMIN_CREDENTIAL_VALIDATOR,
new Persistence(importDir),
new StorageCloudFunctions("project"),
);
imported.import(exportDir, { initiatedBy: "test" });
expect(await readData(imported, "users/B/photo.png")).to.equal("from B");
expect(await readData(imported, "users/A/../B/photo.png")).to.equal("from A");
} finally {
rmSync(exportDir, { recursive: true, force: true });
rmSync(importDir, { recursive: true, force: true });
}
});There was a problem hiding this comment.
Thanks, done in 78daee2: the export/import test now creates both temp directories up front and deletes them in a finally, so they get cleaned up even when an assertion fails. The new | import test cleans up its export directory the same way.
… clean up spec temp dirs getPathSep's character class [\/|\\] also matched a literal |, so a name like a|b\c.txt from an old Windows export was seen as using | and its backslashes were not mapped to /. Check the first / or \ directly instead, and add a test for it. Also remove the temporary export/import directories the new specs create.
Resolve the CHANGELOG conflict by keeping the Storage emulator object-key fix entry alongside post-15.34.1 notes. The Hosting API line was already released, so it stays out of the unreleased log. Co-authored-by: Sahinur <SahinurDEV@users.noreply.github.com>
…ral-object-keys Signed-off-by: SahinurDEV <68515168+SahinurDEV@users.noreply.github.com> # Conflicts: # CHANGELOG.md
…r-literal-object-keys
Description
Fixes #11301.
The Storage emulator evaluates rules against the object name from the request, but it stored and looked up objects under
path.join(bucket, object).path.joincollapses.and..segments, sousers/A/../B/photo.pngwas authorized as a path underusers/A/and then read, overwrote or deletedusers/B/photo.png. Cloud Storage keeps object names literal, so those are two different objects there, and rules tests against the emulator could pass or fail for the wrong reason.StorageLayernow keys objects by${bucket}/${object}with the name unchanged. Bucket names can't contain/, so the key is still unique per bucket and object. The same key is used for uploads, reads, deletes, copies and export/import, so rules and storage agree on the name. (On Windowspath.joinalso turned\in a name into a separator, soa\banda/bcollided; that goes away too.)The one place that relied on the normalized key was import: older exports made on Windows stored names with
\separators (seescripts/storage-emulator-integration/import/windows-emulator-data), and the key was rebuilt with the platform separator. Import now maps those\separators to/, so those exports still load as before.Also added a CHANGELOG entry.
Scenarios Tested
New unit tests in
src/emulator/storage/files.spec.ts:users/A/../B/photo.pngdoesn't overwriteusers/B/photo.png, and both read back their own bytesusers/A/../B/photo.pngorusers/B/./photo.pngreturns not found instead ofusers/B/photo.pngusers/A/../B/photo.pngdoesn't deleteusers/B/photo.pngusers/A/../B/photo.pngdoesn't overwriteusers/B/photo.pngtest-directory/test_nested_upload.jpgandtest_upload.jpgThe five dot-segment tests fail on
mainand pass with this change; the Windows import test passes on both.npx mocha 'src/emulator/**/*.spec.ts': 601 passing, 1 pendingnpx mocha 'src/**/*.spec.{ts,js}': 5884 passing; the only failure isutils datetimeString, which fails the same way onmainbecause my machine's timezone is UTC+6tsc --project tsconfig.compile.json, eslint (no new warnings) and prettier are cleanI didn't run the storage emulator integration suite (
scripts/storage-emulator-integration) locally.Sample Commands
N/A