Skip to content

fix: export importFile and importDirectory function #284

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Feb 17, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 8 additions & 3 deletions packages/ipfs-unixfs-importer/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,8 @@
- [Example](#example)
- [API](#api)
- [const stream = importer(source, blockstore \[, options\])](#const-stream--importersource-blockstore--options)
- [const result = await importContent(content, blockstore \[, options\])](#const-result--await-importcontentcontent-blockstore--options)
- [const result = await importFile(content, blockstore \[, options\])](#const-result--await-importfilecontent-blockstore--options)
- [const result = await importDirectory(content, blockstore \[, options\])](#const-result--await-importdirectorycontent-blockstore--options)
- [const result = await importBytes(buf, blockstore \[, options\])](#const-result--await-importbytesbuf-blockstore--options)
- [const result = await importByteStream(source, blockstore \[, options\])](#const-result--await-importbytestreamsource-blockstore--options)
- [API Docs](#api-docs)
Expand Down Expand Up @@ -97,7 +98,7 @@ When run, metadata about DAGNodes in the created tree is printed until the root:
## API

```js
import { importer, importContent, importBytes } from 'ipfs-unixfs-importer'
import { importer, importFile, importDir, importBytes, importByteStream } from 'ipfs-unixfs-importer'
```

### const stream = importer(source, blockstore \[, options])
Expand All @@ -119,10 +120,14 @@ The `importer` function returns an async iterator takes a source async iterator

The input's file paths and directory structure will be preserved in the [`dag-pb`](https://github.com/ipld/js-dag-pb) created nodes.

### const result = await importContent(content, blockstore \[, options])
### const result = await importFile(content, blockstore \[, options])

A convenience function for importing a single file or directory.

### const result = await importDirectory(content, blockstore \[, options])

A convenience function for importing a directory - note this is non-recursive, to import recursively use the [importer](#const-stream--importersource-blockstore--options) function.

### const result = await importBytes(buf, blockstore \[, options])

A convenience function for importing a single Uint8Array.
Expand Down
8 changes: 6 additions & 2 deletions packages/ipfs-unixfs-importer/src/dag-builder/index.ts
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
import { dirBuilder, DirBuilderOptions } from './dir.js'
import { fileBuilder, FileBuilderOptions } from './file.js'
import errCode from 'err-code'
import type { Directory, File, ImportCandidate, InProgressImportResult } from '../index.js'
import type { Directory, File, FileCandidate, ImportCandidate, InProgressImportResult } from '../index.js'
import type { Blockstore } from 'interface-blockstore'
import type { ChunkValidator } from './validate-chunks.js'
import type { Chunker } from '../chunker/index.js'
Expand Down Expand Up @@ -59,7 +59,7 @@ export function defaultDagBuilder (options: DagBuilderOptions): DAGBuilder {
.join('/')
}

if (entry.content != null) {
if (isFileCandidate(entry)) {
const file: File = {
path: entry.path,
mtime: entry.mtime,
Expand All @@ -84,3 +84,7 @@ export function defaultDagBuilder (options: DagBuilderOptions): DAGBuilder {
}
}
}

function isFileCandidate (entry: any): entry is FileCandidate {
return entry.content != null
}
67 changes: 53 additions & 14 deletions packages/ipfs-unixfs-importer/src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -16,13 +16,21 @@ import type { AwaitIterable } from 'blockstore-core/base'
export type ByteStream = AwaitIterable<Uint8Array>
export type ImportContent = ByteStream | Uint8Array

export interface ImportCandidate {
export interface FileCandidate {
path?: string
content?: ImportContent
content: ImportContent
mtime?: Mtime
mode?: number
}

export interface DirectoryCandidate {
path: string
mtime?: Mtime
mode?: number
}

export type ImportCandidate = FileCandidate | DirectoryCandidate

export interface File {
content: AsyncIterable<Uint8Array>
path?: string
Expand Down Expand Up @@ -180,7 +188,7 @@ export interface ImporterOptions {
chunkValidator?: ChunkValidator
}

export type ImportCandidateStream = AsyncIterable<ImportCandidate> | Iterable<ImportCandidate>
export type ImportCandidateStream = AsyncIterable<FileCandidate | DirectoryCandidate> | Iterable<FileCandidate | DirectoryCandidate>

/**
* The importer creates UnixFS DAGs and stores the blocks that make
Expand Down Expand Up @@ -210,7 +218,7 @@ export type ImportCandidateStream = AsyncIterable<ImportCandidate> | Iterable<Im
* ```
*/
export async function * importer (source: ImportCandidateStream, blockstore: Blockstore, options: ImporterOptions = {}): AsyncGenerator<ImportResult, void, unknown> {
let candidates: AsyncIterable<ImportCandidate> | Iterable<ImportCandidate>
let candidates: AsyncIterable<FileCandidate | DirectoryCandidate> | Iterable<FileCandidate | DirectoryCandidate>

if (Symbol.asyncIterator in source || Symbol.iterator in source) {
candidates = source
Expand Down Expand Up @@ -261,28 +269,59 @@ export async function * importer (source: ImportCandidateStream, blockstore: Blo
}

/**
* `importContent` is similar to `importer` except it accepts a single
* `ImportCandidate` and returns a promise of a single `ImportResult`
* `importFile` is similar to `importer` except it accepts a single
* `FileCandidate` and returns a promise of a single `ImportResult`
* instead of a stream of results.
*
* @example
*
* ```typescript
* import { importOne } from 'ipfs-unixfs-importer'
* import { importFile } from 'ipfs-unixfs-importer'
* import { MemoryBlockstore } from 'blockstore-core'
*
* // store blocks in memory, other blockstores are available
* const blockstore = new MemoryBlockstore()
*
* const input = {
* const input: FileCandidate = {
* path: './foo.txt',
* content: Uint8Array.from([0, 1, 2, 3, 4])
* }
*
* const entry = await importContent(input, blockstore)
* const entry = await importFile(input, blockstore)
* ```
*/
export async function importFile (content: FileCandidate, blockstore: Blockstore, options: ImporterOptions = {}): Promise<ImportResult> {
const result = await first(importer([content], blockstore, options))

if (result == null) {
throw errcode(new Error('Nothing imported'), 'ERR_INVALID_PARAMS')
}

return result
}

/**
* `importDir` is similar to `importer` except it accepts a single
* `DirectoryCandidate` and returns a promise of a single `ImportResult`
* instead of a stream of results.
*
* @example
*
* ```typescript
* import { importDirectory } from 'ipfs-unixfs-importer'
* import { MemoryBlockstore } from 'blockstore-core'
*
* // store blocks in memory, other blockstores are available
* const blockstore = new MemoryBlockstore()
*
* const input: DirectoryCandidate = {
* path: './foo.txt'
* }
*
* const entry = await importDirectory(input, blockstore)
* ```
*/
export async function importContent (content: ImportCandidate, blockstore: Blockstore, options: ImporterOptions = {}): Promise<ImportResult> {
export async function importDirectory (content: DirectoryCandidate, blockstore: Blockstore, options: ImporterOptions = {}): Promise<ImportResult> {
const result = await first(importer([content], blockstore, options))

if (result == null) {
Expand All @@ -299,7 +338,7 @@ export async function importContent (content: ImportCandidate, blockstore: Block
* @example
*
* ```typescript
* import { importOne } from 'ipfs-unixfs-importer'
* import { importBytes } from 'ipfs-unixfs-importer'
* import { MemoryBlockstore } from 'blockstore-core'
*
* // store blocks in memory, other blockstores are available
Expand All @@ -311,7 +350,7 @@ export async function importContent (content: ImportCandidate, blockstore: Block
* ```
*/
export async function importBytes (buf: ImportContent, blockstore: Blockstore, options: ImporterOptions = {}): Promise<ImportResult> {
return await importContent({
return await importFile({
content: buf
}, blockstore, options)
}
Expand All @@ -323,7 +362,7 @@ export async function importBytes (buf: ImportContent, blockstore: Blockstore, o
* @example
*
* ```typescript
* import { importOne } from 'ipfs-unixfs-importer'
* import { importByteStream } from 'ipfs-unixfs-importer'
* import { MemoryBlockstore } from 'blockstore-core'
*
* // store blocks in memory, other blockstores are available
Expand All @@ -338,7 +377,7 @@ export async function importBytes (buf: ImportContent, blockstore: Blockstore, o
* ```
*/
export async function importByteStream (bufs: ByteStream, blockstore: Blockstore, options: ImporterOptions = {}): Promise<ImportResult> {
return await importContent({
return await importFile({
content: bufs
}, blockstore, options)
}