47 lines
1.2 KiB
Markdown
47 lines
1.2 KiB
Markdown
# neural compression
|
|
|
|
## Running locally
|
|
|
|
```
|
|
uv sync --all-extras
|
|
```
|
|
|
|
Example usage:
|
|
|
|
```shell
|
|
# Fetching
|
|
python main.py --debug train --method fetch \
|
|
--dataset enwik9 --data-root /path/to/datasets
|
|
|
|
# Training
|
|
python main.py --debug train --method optuna \
|
|
--dataset enwik9 --data-root /path/to/datasets \
|
|
--model cnn --model-save-path /path/to/optuna-model
|
|
python main.py --debug --results /path/to/results train --method full \
|
|
--dataset enwik9 --data-root /path/to/datasets \
|
|
--model-load-path /path/to/optuna-model --model-save-path /path/to/full-model
|
|
|
|
# Compressing
|
|
python benchmark.py --debug compress \
|
|
--model-load-path /path/to/full-model \
|
|
--input-file inputfile --output-file outputfile
|
|
```
|
|
|
|
Testing compression:
|
|
|
|
```shell
|
|
bash config/download_datasets.sh config/urls.txt /home/tdpeuter/data/ml-inputs
|
|
bash config/generate_csv.sh > config/sub.csv
|
|
bash config/local.sh
|
|
```
|
|
|
|
## Running on the Ghent University HPC
|
|
|
|
See the [Infrastructure docs](https://docs.hpc.ugent.be/infrastructure/#gpu-clusters) for more information about the clusters.
|
|
|
|
```
|
|
module swap cluster/joltik # Specify the (GPU) cluster, {joltik,accelgor,litleo}
|
|
|
|
qsub job.pbs # Submit job
|
|
qstat # Check status
|
|
```
|