How to configure an asset profile with a scraper configuration?

[antanisblinda]

Hey there!
following a discussion on twitter, opening this thread.
I was looking at the premium option for ghostfolio and started with the demo to see how it would work for me.
I noticed that certain (italian) govt bonds cannot be added (example, ISIN IT0001174611, available here).
I understand that i could track it as "valuable" but of course it would mean no automated updates of market value/expiration/etc.
Another option would be a scraper, for example from the URL noted above.

• any hints or documentation reference on how this can be done?
• is this only an option for the self-hosted version or also for the premium tier? I was really looking for a smooth transition without hacking my way into it.

thank you!

[dtslvr]

Hi @antanisblinda

In the self-hosted version of Ghostfolio, you can create custom asset profiles using a scraper configuration. The market data is then crawled periodically. To achieve that, follow these steps:

1. Open the Admin Control panel
2. Go to Market Data
3. Create an asset profile (Add manually...)
4. Edit your asset profile
5. Add the following snippet to Scraper Configuration:

{"selector":".summary-value strong","url":"https://www.borsaitaliana.it/borsa/obbligazioni/mot/btp/scheda/IT0001174611.html?lang=en"}

6. Save the asset profile

[davidecorradini]

How can I add the country(ies) to a MANUAL asset? For example, in this case, it would be 100% Italy. I tried several formats for the "countries" field in the asset, such as ["IT"], ["ITALY"], [{"IT": 1}], but the input is not accepted.

[dtslvr]

This is the format: [{"code": "IT", "weight": 1}]

[gstr3745]

Thank you @dtslvr ! I've been searching for this since a while!
Via trials and errors I found out that syntax for sectors is quite similar [{"name":"Treasury","weight":"1"}]

Are there any lists of accepted values?

[dtslvr]

There is currently no full list of accepted values. However, here are the values extracted from an FTSE All-World ETF that might be helpful (not claiming to be complete).

• Chemicals
• Communication
• Communication Services
• Consumer Cyclical
• Consumer Staples
• Electric Appliances
• Energy
• Financials
• Healthcare
• Industrials
• Information & Communication
• Materials
• Real Estate
• Services
• Technology
• Telecommunications
• Transportation Equipment
• Utilities

[gstr3745]

Thank you very much @dtslvr !

[zaromarco]

Hi , i'm trying to get data with scraper from https://live.euronext.com/nb/product/structured-products/IT0005333742-ETLX/market-information
but with the example provided everytime i get parse error , also on other site
is there a guide to implement scraper - sector - symbol mapping and a guide to fully use the potential of market data?

[PARKOURexe]

The only ticker I'm getting returned is "vanguard-sp-500-etf-tokenized-stock-defichain", which is not the true etf that I'm looking for. I can't seem to pull other financial tickers either, only cryptos

[zaromarco]

very strange...i'm using the latest version self hosted and when i create a new activity and in the name i type VOO i get 6 different etf which one is the same on yahoo finance page

[PARKOURexe]

I am also self-hosted, version 2.61.0

[zaromarco]

that version is from march '24 , i'm using 2.77.1
maybe is worth try to upgrade and see what will show

[gstr3745]

Market data fetch is broken for previous versions (#3276 , corrected with #3286 ): you should upgrade at least to 2.69.0 (if I recall it right).
Anyway, latest is working smoothely

[colorcube]

Because related:

• what data should be put into the field "Symbol Mapping"
• what does Benchmark do?
• can the imported "Historical Data" be deleted (in case of wrong data for example)?

[gstr3745]

you shuld populate "symbol mapping" to get the sector and countries data (e.g. from trackinsight)

the check on "benchmark" lets you graph the asset on your portfolio overview, as far as I understood: so you could compare easily your portfolio with the single asset or index

as far as I know, you could overwrite historical data with manual csv import, but I'm not aware if it's possible to delete previous scraped or gathered data, I'm sorry

This is what I assumed, using ghostfolio: I hope I'm not misunderstanding anything.

[colorcube]

you shuld populate "symbol mapping" to get the sector and countries data (e.g. from trackinsight)

How? what's the format?

[colorcube]

For ISIN IE00BJ0KDQ92 the trackinsight page is
https://www.trackinsight.com/de/fund/24869934

In the source code you find the code QQQ for TrackinsightDataEnhancerService.
Which means the field "symbol mapping" should be:

{"QQQ":"24869934"}

But I have no idea how to trigger the service. "Gather * Data" menu options has no effect.

[gstr3745]

If I recall right, it should be:
{"TRACKINSIGHT":"24869934"}

I cannot try this right now on my instance, I hope this will work

[colorcube]

Right, still no luck. I'll look into it.

Thanks for your help!