downloadTool

massivedl

Download a large list of files in parallel.

Usage

Create a .csv file with the downloads

1
2
3
4
5
filename,url
0.png,https://placehold.it/100x100
1.png,https://placehold.it/100x101
2.png,https://placehold.it/100x102
...

Assuming the file was named data.csv we can download the files using

1
massivedl -p 10 -i data.csv -s 1 -o downloads

Command line parameters

1
2
3
4
-p <int> (default=10)          : Maximum number of parallel requests
-s <int> (default=0) : Number of skipped lines from input csv
-i <str> : Input csv file with the list of urls
-o <str> (default='downloads') : Directory to place the downloads

Stop and continue later

You can stop and continue downloading later.
Press Ctrl+C then you will have the following dialog.

1
2
3
4
5
6
7
...
Do you want to save progress? [Y/n]: yes

Progress has been saved!
Use the following command to continue downloading

massivedl --load /path/to/savedfile.save

Use Cases

With this tool I was able to download about 1.5 million images (~60GB) for a machine learning project.

leaflet加载ArcGis服务

1.写在前面

在前端中leaflet天地图使用还是比较多的,可以使用不同坐标系,加载不同的服务数据

2.记录项目使用

使用leaflet加载arcGIS的底图服务,需要先引入 esri-leaflet

1
yarn add esri-leaflet

还在用npm? 你真out,什么你用的cnpm? 哦,随便

1
import * as esri from 'esri-leaflet/dist/esri-leaflet.js';

太简单,跳过

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
init() {
var lMap = L.map("CloudChart", {
center: [29.59, 105.29],
zoom: 10,
layers: [],
maxZoom: 15,
minZoom:3,
zoomControl: true
});
L.control.zoom({
zoomInTitle: "放大",
zoomOutTitle: "缩小"
}).addTo(lMap);
var dynamicLayer = esri.dynamicMapLayer({
url: arcgisBackgroundService.url,
opacity: 0.8,
f:'json'
});
lMap.addLayer(dynamicLayer);
}

初始化函数及基本的配置

3.总结

esri-leaflet是一个专门为leaflet地图加载arcGIS服务的插件,里面封装了很多种图层,详细可以使用console.log()或查看npm包的源码,使用上比较简单,也只有几行代码就实现了服务的加载。
天地图的特色就是代码量少,很少,非常少,快,很快,非常快

附上esri的官方地址吧。
Esri Leaflet