Process getting Killed

[preritt]

We are trying to run https://github.com/dessa-public/fake-voice-detection and got the following error and the process is getting killed. Not sure if you could help with this.(fake_voice)

neutigers@tensorbook:~/Desktop/fake_voice_detection/fake-voice-detection/code$ python main.py
WARNING:tensorflow:
The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:
  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
  * https://github.com/tensorflow/addons
  * https://github.com/tensorflow/io (for I/O related ops)
If you depend on functionality not listed there, please file an issue.Using TensorFlow backend.
/home/neutigers/anaconda3/envs/fake_voice/lib/python3.7/site-packages/sklearn/externals/joblib/__init__.py:15: DeprecationWarning: sklearn.externals.joblib is deprecated in 0.21 and will be removed in 0.23. Please import this functionality directly from joblib, which can be installed with: pip install joblib. If this warning is raised when loading pickled models, you may need to re-serialize those models with scikit-learn 0.21+.
  warnings.warn(msg, category=DeprecationWarning)
/home/neutigers/anaconda3/envs/fake_voice/lib/python3.7/site-packages/foundations_contrib/config_manager.py:90: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  return yaml.load(file)
/home/neutigers/anaconda3/envs/fake_voice/lib/python3.7/site-packages/foundations_contrib/cli/config_listing.py:41: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  return yaml.load(file.read())
/home/neutigers/anaconda3/envs/fake_voice/lib/python3.7/site-packages/foundations_internal/config/execution.py:18: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  schema = yaml.load(file.read())
loading the preprocessed training, validation and test data.
Killed
GitHubGitHub
dessa-public/fake-voice-detection
Using temporal convolution to detect Audio Deepfakes - dessa-public/fake-voice-detection

[ranasac19878]

Thanks for posting this issue, I am looking at it and will post a solution soon.

[ranasac19878]

It looks like your computer's memory is running out which is why the process is getting killed. This is happening during loading the preprocessed data which is huge and runs over 25 GB so make sure you have that much RAM available. Next time, please monitor your RAM while the program is running and let me know if you memory is not the problem.

[preritt]

Thanks. My computer's memory is 32 GB but it could be other processes are running in the background. Is there a way to old the data with limited ram like breaking it apart or the only solution is to use computer with more memory?

[ranasac19878]

You can download the raw data from this website: https://datashare.is.ed.ac.uk/handle/10283/3336
For this project, we have used the logical access part of this dataset. After downloading the dataset, you will need to run an extra step of preprocessing the data. You can refer to unlabelled_inference.py file to see what function you need to run in order to preprocess the audio files. Here is the snippet below:

processed_data = preprocess_from_ray_parallel_inference(data_dir, mode, use_parallel=False)

But this whole thing will take you a few hours. My suggenstion is that you can write a seperate python script, load the traiining, validation and test data one by one and save a subset of them onto your local computer. In this case, you will be decreasing the data for neural network which is not good.

Another thing is that you can save the preprocessed data in HDF5 format on your disk which can be read directly by neural net. This is the most memory efficient solution.

Let me know if these ideas help.

[yzslry]

The link to download the data in this project seems to be invalid. Can you provide the data or link in the project?