init depot renew
This commit is contained in:
@@ -0,0 +1,73 @@
|
||||
|
||||
|
||||
# UsersHub
|
||||
VERSION=2.4.7
|
||||
cd
|
||||
wget https://github.com/PnX-SI/UsersHub/archive/$VERSION.zip
|
||||
unzip $VERSION.zip
|
||||
rm $VERSION.zip
|
||||
sudo rm -R ~/usershub_old
|
||||
mv /home/`whoami`/usershub/ /home/`whoami`/usershub_old/
|
||||
mv UsersHub-$VERSION /home/`whoami`/usershub/
|
||||
cp /home/`whoami`/usershub_old/config/config.py /home/`whoami`/usershub/config/config.py
|
||||
cp /home/`whoami`/usershub_old/config/settings.ini /home/`whoami`/usershub/config/settings.ini
|
||||
cd usershub
|
||||
./install_app.sh
|
||||
|
||||
# Monitoring
|
||||
VERSION=1.0.3
|
||||
cd
|
||||
wget https://github.com/PnX-SI/gn_module_monitoring/archive/$VERSION.zip
|
||||
unzip $VERSION.zip
|
||||
rm $VERSION.zip
|
||||
sudo rm -R ~/gn_module_monitoring_old
|
||||
mv ~/gn_module_monitoring/ ~/gn_module_monitoring_old/
|
||||
mv ~/gn_module_monitoring-$VERSION ~/gn_module_monitoring
|
||||
|
||||
# Export
|
||||
VERSION=1.7.2
|
||||
cd
|
||||
wget https://github.com/PnX-SI/gn_module_export/archive/$VERSION.zip
|
||||
unzip $VERSION.zip
|
||||
rm $VERSION.zip
|
||||
sudo rm -R ~/gn_module_export_old
|
||||
mv ~/gn_module_export/ ~/gn_module_export_old/
|
||||
mv ~/gn_module_export-$VERSION ~/gn_module_export
|
||||
|
||||
# ~/gn_module_export_old/backend/templates/swagger
|
||||
# ZH
|
||||
VERSION=1.4.0
|
||||
cd
|
||||
wget https://github.com/PnX-SI/gn_module_ZH/archive/$VERSION.zip
|
||||
unzip $VERSION.zip
|
||||
rm $VERSION.zip
|
||||
sudo rm -R ~/gn_module_ZH_old
|
||||
mv ~/gn_module_ZH/ ~/gn_module_ZH_old/
|
||||
mv ~/gn_module_ZH-$VERSION ~/gn_module_ZH
|
||||
|
||||
# GeoNature
|
||||
VERSION=2.15.4
|
||||
wget https://github.com/PnX-SI/GeoNature/archive/$VERSION.zip
|
||||
unzip $VERSION.zip
|
||||
rm $VERSION.zip
|
||||
sudo rm -R ~/geonature_old
|
||||
mv ~/geonature/ ~/geonature_old/
|
||||
mv ~/GeoNature-$VERSION ~/geonature/
|
||||
cd ~/geonature
|
||||
./install/migration/migration.sh 2>&1 | tee install/migration/migration.log
|
||||
|
||||
1. Remove API_TAXHUB
|
||||
2. Modifier taxhub_url IN /geonature/backend/media/mobile/occtax/settings.json
|
||||
3. geonature db downgrade taxhub-standalone@base
|
||||
4. Check media taxhub IN GeoNature THEN rm ~/taxhub
|
||||
5. Modifier config Apache TaxHub
|
||||
``` sh
|
||||
sudo nano /etc/apache2/conf-available/taxhub.conf
|
||||
|
||||
# Cas où TaxHub et GeoNature sont sur le même sous-domaine
|
||||
RewriteEngine on
|
||||
RewriteRule "^/taxhub/static/medias/(.+)" "/geonature/api/medias/taxhub/$1" [R,L]
|
||||
```
|
||||
~6. geonature monitorings update_module_available_permissions
|
||||
~7. RUN data/upgrade_modules_permissions.sql
|
||||
8. Update VIEWS Monitorings
|
||||
@@ -0,0 +1,128 @@
|
||||
|
||||
# Monitoring
|
||||
VERSION=1.1.0
|
||||
cd
|
||||
wget https://github.com/PnX-SI/gn_module_monitoring/archive/$VERSION.zip
|
||||
unzip $VERSION.zip
|
||||
rm $VERSION.zip
|
||||
sudo rm -R ~/gn_module_monitoring_old
|
||||
mv ~/gn_module_monitoring/ ~/gn_module_monitoring_old/
|
||||
mv ~/gn_module_monitoring-$VERSION ~/gn_module_monitoring
|
||||
|
||||
# Export
|
||||
VERSION=1.8.0
|
||||
cd
|
||||
wget https://github.com/PnX-SI/gn_module_export/archive/$VERSION.zip
|
||||
unzip $VERSION.zip
|
||||
rm $VERSION.zip
|
||||
sudo rm -R ~/gn_module_export_old
|
||||
mv ~/gn_module_export/ ~/gn_module_export_old/
|
||||
mv ~/gn_module_export-$VERSION ~/gn_module_export
|
||||
|
||||
# ~/gn_module_export_old/backend/templates/swagger
|
||||
# DASHBOARD
|
||||
VERSION=1.6.0
|
||||
cd
|
||||
wget https://github.com/PnX-SI/gn_module_dashboard/archive/$VERSION.zip
|
||||
unzip $VERSION.zip
|
||||
rm $VERSION.zip
|
||||
sudo rm -R ~/gn_module_dashboard_old
|
||||
mv ~/gn_module_dashboard/ ~/gn_module_dashboard_old/
|
||||
mv ~/gn_module_dashboard-$VERSION ~/gn_module_dashboard
|
||||
|
||||
# GeoNature
|
||||
VERSION=2.16.2
|
||||
wget https://github.com/PnX-SI/GeoNature/archive/$VERSION.zip
|
||||
unzip $VERSION.zip
|
||||
rm $VERSION.zip
|
||||
sudo rm -R ~/geonature_old
|
||||
mv ~/geonature/ ~/geonature_old/
|
||||
mv ~/GeoNature-$VERSION ~/geonature/
|
||||
cd ~/geonature
|
||||
./install/migration/migration.sh 2>&1 | tee install/migration/migration.log
|
||||
|
||||
|
||||
|
||||
# LOG WARNING
|
||||
Considering dependency proxy for proxy_http:
|
||||
Module proxy already enabled
|
||||
Module proxy_http already enabled
|
||||
Considering dependency filter for deflate:
|
||||
Module filter already enabled
|
||||
Module deflate already enabled
|
||||
AH00112: Warning: DocumentRoot [/home/geonatureadmin/geonature/frontend/dist] does not exist
|
||||
Syntax OK
|
||||
Set API_ENDPOINT to https://geonature.cen-isere.fr/geonature/api in frontend configuration file...
|
||||
Mise à jour des fichiers de configuration frontend et rebuild du frontend…
|
||||
Génération de la configuration du frontend :
|
||||
|
||||
# LOG ERROR
|
||||
Mise à jour de la base de données…
|
||||
/home/geonatureadmin/geonature/backend/venv/lib/python3.10/site-packages/alembic/config.py:560: DeprecationWarning: No path_separator found in configuration; falling back to legacy splitting on spaces/commas for version_locations. Consider adding path_separator=os to Alembic config.
|
||||
util.warn_deprecated(
|
||||
/home/geonatureadmin/geonature/backend/geonature/migrations/versions/7dfd0a813f86_insert_inpn_sensitivity_referential.py:14: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
|
||||
from distutils.util import strtobool
|
||||
/home/geonatureadmin/geonature/backend/venv/lib/python3.10/site-packages/alembic/config.py:560: DeprecationWarning: No path_separator found in configuration; falling back to legacy splitting on spaces/commas for version_locations. Consider adding path_separator=os to Alembic config.
|
||||
util.warn_deprecated(
|
||||
INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
|
||||
INFO [alembic.runtime.migration] Will assume transactional DDL.
|
||||
/home/geonatureadmin/geonature/backend/geonature/migrations/versions/7dfd0a813f86_insert_inpn_sensitivity_referential.py:14: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
|
||||
from distutils.util import strtobool
|
||||
ERROR [flask_migrate] Error: Can't locate revision identified by 'd85b87a1ca62'
|
||||
/home/geonatureadmin/geonature/backend/venv/lib/python3.10/site-packages/utils_flask_sqla/commands.py:89: DeprecationWarning: The 'db' attribute is deprecated and will be removed in Flask-SQLAlchemy 3.1. The extension is registered directly as 'app.extensions["sqlalchemy"]'.
|
||||
db = current_app.extensions["sqlalchemy"].db
|
||||
/home/geonatureadmin/geonature/backend/venv/lib/python3.10/site-packages/alembic/config.py:560: DeprecationWarning: No path_separator found in configuration; falling back to legacy splitting on spaces/commas for version_locations. Consider adding path_separator=os to Alembic config.
|
||||
util.warn_deprecated(
|
||||
/home/geonatureadmin/geonature/backend/geonature/migrations/versions/7dfd0a813f86_insert_inpn_sensitivity_referential.py:14: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
|
||||
from distutils.util import strtobool
|
||||
Traceback (most recent call last):
|
||||
File "/home/geonatureadmin/geonature/backend/venv/lib/python3.10/site-packages/alembic/script/base.py", line 217, in _catch_revision_errors
|
||||
yield
|
||||
File "/home/geonatureadmin/geonature/backend/venv/lib/python3.10/site-packages/alembic/script/base.py", line 285, in get_all_current
|
||||
return cast(Set[Script], self.revision_map._get_all_current(id_))
|
||||
File "/home/geonatureadmin/geonature/backend/venv/lib/python3.10/site-packages/alembic/script/revision.py", line 1531, in _get_all_current
|
||||
top_revs = set(self.get_revisions(id_))
|
||||
File "/home/geonatureadmin/geonature/backend/venv/lib/python3.10/site-packages/alembic/script/revision.py", line 542, in get_revisions
|
||||
return sum([self.get_revisions(id_elem) for id_elem in id_], ())
|
||||
File "/home/geonatureadmin/geonature/backend/venv/lib/python3.10/site-packages/alembic/script/revision.py", line 542, in <listcomp>
|
||||
return sum([self.get_revisions(id_elem) for id_elem in id_], ())
|
||||
File "/home/geonatureadmin/geonature/backend/venv/lib/python3.10/site-packages/alembic/script/revision.py", line 565, in get_revisions
|
||||
return tuple(
|
||||
File "/home/geonatureadmin/geonature/backend/venv/lib/python3.10/site-packages/alembic/script/revision.py", line 566, in <genexpr>
|
||||
self._revision_for_ident(rev_id, branch_label)
|
||||
File "/home/geonatureadmin/geonature/backend/venv/lib/python3.10/site-packages/alembic/script/revision.py", line 637, in _revision_for_ident
|
||||
raise ResolutionError(
|
||||
alembic.script.revision.ResolutionError: No such revision or branch 'd85b87a1ca62'
|
||||
|
||||
The above exception was the direct cause of the following exception:
|
||||
|
||||
Traceback (most recent call last):
|
||||
File "/home/geonatureadmin/geonature/backend/venv/bin/geonature", line 7, in <module>
|
||||
sys.exit(main())
|
||||
File "/home/geonatureadmin/geonature/backend/venv/lib/python3.10/site-packages/click/core.py", line 1161, in __call__
|
||||
return self.main(*args, **kwargs)
|
||||
File "/home/geonatureadmin/geonature/backend/venv/lib/python3.10/site-packages/click/core.py", line 1082, in main
|
||||
rv = self.invoke(ctx)
|
||||
File "/home/geonatureadmin/geonature/backend/venv/lib/python3.10/site-packages/click/core.py", line 1697, in invoke
|
||||
return _process_result(sub_ctx.command.invoke(sub_ctx))
|
||||
File "/home/geonatureadmin/geonature/backend/venv/lib/python3.10/site-packages/click/core.py", line 1697, in invoke
|
||||
return _process_result(sub_ctx.command.invoke(sub_ctx))
|
||||
File "/home/geonatureadmin/geonature/backend/venv/lib/python3.10/site-packages/click/core.py", line 1443, in invoke
|
||||
return ctx.invoke(self.callback, **ctx.params)
|
||||
File "/home/geonatureadmin/geonature/backend/venv/lib/python3.10/site-packages/click/core.py", line 788, in invoke
|
||||
return __callback(*args, **kwargs)
|
||||
File "/home/geonatureadmin/geonature/backend/venv/lib/python3.10/site-packages/click/decorators.py", line 33, in new_func
|
||||
return f(get_current_context(), *args, **kwargs)
|
||||
File "/home/geonatureadmin/geonature/backend/venv/lib/python3.10/site-packages/flask/cli.py", line 400, in decorator
|
||||
return ctx.invoke(f, *args, **kwargs)
|
||||
File "/home/geonatureadmin/geonature/backend/venv/lib/python3.10/site-packages/click/core.py", line 788, in invoke
|
||||
return __callback(*args, **kwargs)
|
||||
File "/home/geonatureadmin/geonature/backend/venv/lib/python3.10/site-packages/utils_flask_sqla/commands.py", line 97, in autoupgrade
|
||||
current_heads = set(map(lambda rev: rev.revision, script.get_all_current(current_heads)))
|
||||
File "/home/geonatureadmin/geonature/backend/venv/lib/python3.10/site-packages/alembic/script/base.py", line 284, in get_all_current
|
||||
with self._catch_revision_errors():
|
||||
File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
|
||||
self.gen.throw(typ, value, traceback)
|
||||
File "/home/geonatureadmin/geonature/backend/venv/lib/python3.10/site-packages/alembic/script/base.py", line 249, in _catch_revision_errors
|
||||
raise util.CommandError(resolution) from re
|
||||
alembic.util.exc.CommandError: Can't locate revision identified by 'd85b87a1ca62'
|
||||
@@ -0,0 +1,68 @@
|
||||
from pycen import con_gn
|
||||
import pycen
|
||||
import pandas as pd
|
||||
import os
|
||||
|
||||
def get_taxref(con):
|
||||
sql = '''SELECT cd_nom,cd_ref,nom_complet,lb_nom FROM taxonomie.taxref;'''
|
||||
return pd.read_sql_query(sql,con)
|
||||
|
||||
def get_cd_nom(con,lb):
|
||||
sql = '''SELECT cd_nom,cd_ref,nom_complet,lb_nom FROM taxonomie.taxref
|
||||
WHERE nom_complet='{label}' or nom_complet ilike '%%{label}%%';'''.format(label=lb)
|
||||
return pd.read_sql_query(sql,con)
|
||||
|
||||
|
||||
dict_taxo = {
|
||||
'Brachythecium rutabulum (Hedw.) Schimp. var. rutabulum':434653,
|
||||
'Hypnum cupressiforme Hedw. var. cupressiforme':434662,
|
||||
'Orthotrichum affine Brid.':5014,
|
||||
'Plagiomnium cuspidatum ( Hedw.) T.J.Kop.':4934,
|
||||
'Leucodon sciuroides (Hedw.) Schwägr var. sciuroides':None,
|
||||
'Brachythecium albicans (Hedw. ) Schimp.':5807,
|
||||
'Phascum cuspidatum Hedw. var. cuspidatum':434590,
|
||||
'Syntrichia ruralis (Hedw.) F.Weber & D.Mohr var. ruralis':434602,
|
||||
'Brachythecium salebrosum (Hoffm. ex F.Weber & D.Mohr) Schimp. nom. cons.':5828,
|
||||
'Lunularia cruciata (L.) Lindb.':6167,
|
||||
'Rhynchostegium megapolitanum (Blandow ex F.Weber & D.Mohr) Schimp.':5908,
|
||||
'Orthotrichum cupulatum Hoffm. ex Brid. var. riparium Huebener':434606,
|
||||
'Syntrichia montana Nees var. montana':434600,
|
||||
'Abietinella abietina (Hedw.) M.Fleisch. var. abietina':434642,
|
||||
'Ceratodon purpureus (Hedw.) Brid. subsp. purpureus':434562,
|
||||
'Hedwigia ciliata (Hedw.) P.Beauv. var. leucophaea Bruch & Schimp.':434612,
|
||||
'Hypnum cupressiforme Hedw. var. filiforme Brid.':6046,
|
||||
'Hedwigia ciliata (Hedw.) P.Beauv. var. ciliata':434611,
|
||||
'Orthotrichum speciosum Nees var. speciosum':None,
|
||||
'Brachytheciastrum velutinum (Hedw.) Ignatov & Huttunen var. velutinum':434659
|
||||
}
|
||||
dict_taxo2 = {
|
||||
'Leucodon sciuroides (Hedw.) Schwägr var. sciuroides':5104,
|
||||
'Orthotrichum speciosum Nees var. speciosum':5041,
|
||||
}
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
PATH = '/media/cgeier/Disk2/tmp'
|
||||
file = '(RNN Platière) Inventaire Bryophytes L93.csv'
|
||||
file_out = '(RNN Platière) Inventaire Bryophytes L93 (correct cdnom).csv'
|
||||
df = pd.read_csv(os.path.join(PATH,file),encoding='Windows-1252',sep=';')
|
||||
tax = get_taxref(con_gn)
|
||||
|
||||
for n in df.NOM_COMPLET.unique():
|
||||
|
||||
res = get_cd_nom(con_gn,n)
|
||||
df.loc[df.NOM_COMPLET==n,'CD_NOM2'] = res.cd_nom.values[0] if not res.empty else None
|
||||
if res.shape[0]>1:
|
||||
print(n)
|
||||
|
||||
df.CD_NOM2 = df.CD_NOM2.astype('Int64')
|
||||
cd_isna = df.CD_NOM2.isna()
|
||||
df.loc[cd_isna,'CD_NOM2'] = df.loc[cd_isna,'NOM_COMPLET'].replace(dict_taxo).astype('Int64')
|
||||
cd_isna = df.CD_NOM2.isna()
|
||||
df.loc[cd_isna,'remarque_obs'] = "Nom complet d'origine : "+df[cd_isna].NOM_COMPLET
|
||||
df.loc[cd_isna,'CD_NOM2'] = df.loc[cd_isna,'NOM_COMPLET'].replace(dict_taxo2).astype('Int64')
|
||||
df.CD_NOM2 = df.CD_NOM2.astype(int)
|
||||
df.to_csv(os.path.join(PATH,file_out),encoding='Windows-1252',sep=';',index=False)
|
||||
|
||||
|
||||
@@ -0,0 +1,428 @@
|
||||
#!/usr/bin/env python3
|
||||
# -*- coding: UTF-8 -*-
|
||||
|
||||
import geopandas as gpd
|
||||
from shapely import wkt
|
||||
from shapely.ops import nearest_points,Polygon,Point,LineString
|
||||
from os import path,listdir
|
||||
|
||||
PATH = '/home/colas/Documents/9_PROJETS/6_GEONATURE/MIGRATION/PLATIERE/4_JDD_Faune_Mailles_A_Importer'
|
||||
FILE_DATA = '(CNPE) Chrono-inventaires entomologiques Orthopteres.csv'
|
||||
FILE_DATA = '(CNPE) Observations Opportunistes Orthopteres.csv'
|
||||
FILE_EF_0 = 'Ensembles_Fonctionnels_Platière_Lambert93.shp'
|
||||
FILE_GRID = 'grillegographiquecomplte/Grille_Geographique_Lignes_Complete_WGS84.shp'
|
||||
dictef_cols = {
|
||||
'code_ensemble':'code_ensem',
|
||||
**dict.fromkeys(['ensemble_fonctionnel','nom_ensemble'],'nom_ensemb'),
|
||||
}
|
||||
ef0 = gpd.read_file(path.join(PATH,'..',FILE_EF_0),encoding='UTF-8')
|
||||
ef1 = gpd.read_file(path.join(PATH,'..','ensemblefonctionnelscomplte','ensemble fonc baix.TAB'),encoding='UTF-8')
|
||||
ef2 = gpd.read_file(path.join(PATH,'..','ensemblefonctionnelscomplte','ensemble fonc donzère.TAB'),encoding='UTF-8')
|
||||
ef3 = gpd.read_file(path.join(PATH,'..','ensemblefonctionnelscomplte','Ensembles fonctionnels montélimar.TAB'),encoding='UTF-8')
|
||||
ef4 = gpd.read_file(path.join(PATH,'..','ensemblefonctionnelscomplte','Secteurs obs oiseaux montélimar.TAB'),encoding='UTF-8')
|
||||
for e in [ef1,ef2,ef3,ef4]:
|
||||
e.columns = e.columns.str.lower()
|
||||
e.rename(columns=dictef_cols,inplace=True)
|
||||
if e.crs.srs.lower() != 'epsg:2154':
|
||||
e.to_crs('epsg:2154',inplace=True)
|
||||
ef = (gpd.pd.concat([ef0,ef1,ef2,ef3,ef4])
|
||||
.astype({'code_ensem':str})
|
||||
.sort_values(['nom_ensemb'],ignore_index=True)
|
||||
.drop_duplicates(ignore_index=True))
|
||||
ef.to_file(path.join(PATH,'..','ensemblefonctionnelscomplte','Ensembles_Fonctionnels_Platière_Lambert93_ALL.geojson'),driver='GeoJSON')
|
||||
data = gpd.pd.read_csv(path.join(PATH,FILE_DATA),sep=";",encoding='Windows 1252') # ['ASCII','Windows 1252']
|
||||
|
||||
|
||||
dict_col = {
|
||||
**dict.fromkeys(['longitude autre'],'Long autre'),
|
||||
**dict.fromkeys(['latitude autre'],'Lat autre'),
|
||||
**dict.fromkeys([' Longitude grade ','LONGITUDE'],'Longitude grade'),
|
||||
**dict.fromkeys([' Latitude grade ','LATITUDE'],'Latitude grade'),
|
||||
**dict.fromkeys(['latitude autre'],'Lat autre'),
|
||||
**dict.fromkeys(['Numéro relevé','Numéro de relevé'],'Numéro'),
|
||||
}
|
||||
|
||||
dict_ef = {
|
||||
'Prairie ancienne pépinière CNR':'Ancienne pépinière CNR',
|
||||
'Marais méandre des Oves':'Marais des Oves',
|
||||
'Prairie des Iles (Péage)':'Prairie des Iles',
|
||||
'Vieux Rhône Péage amont seuil':'Vieux Rhône amont seuil Sablons',
|
||||
'Lône de la Platière':'Lone Platière',
|
||||
'Banc Arcoules (banc1)':'Banc Arcoule (banc 1)',
|
||||
'Plaine St Marice rive gauche canal':'Plaine St Maurice rive gauche canal',
|
||||
'Casiers île des Graviers':'Casiers aval observatoire',
|
||||
'Prairie violettes (vers Marais des Oves)':'Prairie Violettes',
|
||||
'Forêt nord des Oves':'Forêt Nord Oves',
|
||||
'Digue neuve et entonnement de la lône':'Digue neuve et entonnement',
|
||||
'Marais méandre des Oves':'Marais des Oves',
|
||||
'Banc et ilots aval seuil Peyraud':'banc et ilots aval seuil Peyraud',
|
||||
'Casiers banc 3':'casiers banc 3',
|
||||
'Casiers RD Limony/Serrières':'Casiers RD Limony Serrieres',
|
||||
'Forêt Pointe sud':'Forêt pointe Sud',
|
||||
'Gravière des Rotissots (CORA)':'Gravière des Rotissots(CORA)',
|
||||
'Lone des îles':'Boisements MICRONAT',
|
||||
"Lône de l'Ilon":'Lône Ilon',
|
||||
'Lône du Bayard (Donzère)':'Lône du Bayard',
|
||||
'Lône du Noyer nord':'Lône Noyer Nord',
|
||||
'Lône du Noyer sud':'Lône Noyer Sud',
|
||||
'Lone de la Roussette':'Lône de la Roussette',
|
||||
"Plan d'eau St Pierre de Bœuf":"Plan d'eau St Pierre de Boeuf",
|
||||
'Prairie SW +extS+ zone embroussaillée':'Prairie SW + ext Sud + zone embrousaillée',
|
||||
'Remblais Ecluse Sablons':'Remblais usine ecluse',
|
||||
'Roselière et marais St-Maurice':'Roselière et marais St Maurice',
|
||||
'Vieux Rhône Péage aval seuil Sablons':'Vieux Rhône aval Sablons',
|
||||
'Zone cultivée nord':'Zone cultivée Nord',
|
||||
'Contre canal RD canal Péage':'Contre-canal RD Péage',
|
||||
'Aval Ruisseau de Limony':'Foret Limony',
|
||||
'Forêts des grandes Oves':'Forêt des Grandes Oves',
|
||||
'Ile du Noyer nord':'Ile Noyer Nord',
|
||||
"Lone de l'ile de Peyraud":"Lône de l'ile de Peyraud",
|
||||
'Casiers banc 3':'casiers banc 3',
|
||||
'Foret Ile Bugnon':'Forêt Ile Bugnon',
|
||||
'Graviere des Rotissots (Commune)':'Gravière Rotissots (Commune)',
|
||||
'Plaine de St Pierre de Bœuf':'Plaine St Pierre de Boeuf',
|
||||
'Plaine nord Serrières':'Plaine Nord Serrières',
|
||||
'Zone cultivée sud':'Zone cultivée Sud',
|
||||
'Casiers Boussarde':'casiers Boussarde',
|
||||
'Casiers aval seuil Peyraud RG':'Casiers aval seil Peyraud RG',
|
||||
'Contre-canal rive gauche Péage':'Contre-canal RG Péage',
|
||||
'Ile du Noyer sud + boisement ouest':'Ile Noyer Sud + boisement ouest',
|
||||
'Lone de la Barcasse':'Lône de la Barcasse',
|
||||
'Ruisseau Dolon':'Ruisseau Sanne Dolon', # c'est bon !?
|
||||
'la Payre, cours aval + cc RD amont':'La Payre, cours aval + CC RD amont',
|
||||
'Lône de la quarantaine':'Lône de la Quarantaine',
|
||||
"Plan d'eau des Blaches":"plan d'eau des Blaches",
|
||||
'Lone du Gouvernement':'Lône du Gouvernement',
|
||||
'Anc. Gravière CNR Amont RCC Donzère':'Anc Gravière CNR Amont RCC', # ?
|
||||
'Zone réhabilitée traitement granulat RCC Baix':'Zone réhabilitée traitement granulat', # ?
|
||||
# 'Banc 1 RCC Donzère',
|
||||
# 'Butte (Arras sur Rhône)',
|
||||
# 'Canaux et ruisseaux Ancone',
|
||||
# 'Canaux et ruisseaux Petits Robins',
|
||||
# 'Casiers Ile du Chambon',
|
||||
# 'Etang de la Gare',
|
||||
# 'Forêt Limony (hors RN)',
|
||||
# 'Graviere Ancone',
|
||||
# 'Gravières Cruas',
|
||||
# 'Le Petit-Rhône de la Voulte/Livron',
|
||||
# 'Lone Ancone',
|
||||
# 'Prairie ile du Chambon',
|
||||
# 'Prairie sèche "Picard", St Gervais/Roubion',
|
||||
# 'Rivière Roubion',
|
||||
# 'Ruisseau de Limony (gorges)',
|
||||
# 'Ruisseau de la Riaille',
|
||||
# 'Ruisseau du Lumas',
|
||||
# 'Ruisseau ile du Chambon',
|
||||
|
||||
|
||||
# 'Casiers RCC Gervans':,
|
||||
# 'Forêt Limony (hors RN)':,
|
||||
# 'coteaux rive droite de St Pierre à Champagne':,
|
||||
# 'Butte (Arras sur Rhône)':,
|
||||
# 'Prairie ile du Chambon':,
|
||||
# 'Ravin de Serrières':,
|
||||
# 'ruisseau de Peyraud (gorges)':,
|
||||
# 'Rivière Roubion':'Roubion',
|
||||
# 'Banc 2 RCC Donzère',
|
||||
# 'Canaux et ruisseaux Petits Robins',
|
||||
# 'Contre canal RG Planaris',
|
||||
# 'Contre canaux Cruas',
|
||||
# 'Etang Chanson (Chavanay)',
|
||||
# 'Etangs Verlieux',
|
||||
# 'Forêt alluviale Roubion',
|
||||
# 'Grand val (Chavanay)',
|
||||
# 'Graviere Ancone',
|
||||
# 'Lac des Sauzets',
|
||||
# 'Le Petit-Rhône de la Voulte/Livron',
|
||||
# 'Lone des Petits Robins',
|
||||
# 'Marais "de Printegarde"',
|
||||
# "Plan d'eau loisir Le Pouzin",
|
||||
# 'Prairie sèche Pierrelatte',
|
||||
# 'Retenue de Baix-Logis Neuf',
|
||||
# 'Rivière Drôme aval ramières',
|
||||
# 'Rivière Roubion',
|
||||
# 'Ruisseau de Limony (gorges)',
|
||||
# 'Ruisseau ile du Chambon',
|
||||
# 'Vieux rhone de baix',
|
||||
# 'Vieux-Rhône Donzère --> confluent Ardèche',
|
||||
# 'ruisseau de Peyraud (gorges)',
|
||||
|
||||
}
|
||||
|
||||
|
||||
def remove_special_char(obj,space=False):
|
||||
dict_char = {
|
||||
r'[’]':"'",
|
||||
r'[àáâãäå]':'.',
|
||||
r'[èéêë]':'.',
|
||||
r'[ìíîï]':'.',
|
||||
r'[òóôõö]':'.',
|
||||
r'[ùúûü]':'.',
|
||||
r'[ -]':"",
|
||||
r'[–?|!^]':"."
|
||||
}
|
||||
if space:
|
||||
dict_char = {**dict_char, **{r'[ ]':""}}
|
||||
return obj.replace(dict_char,regex=True)
|
||||
|
||||
|
||||
|
||||
def exend_lines(line,geom):
|
||||
if line.geom_type == 'MultiLineString':
|
||||
mini = gpd.GeoSeries(list(line.geoms),crs=2154).distance(geom).min()
|
||||
line = [l for l in line.geoms if l.distance(geom) == mini][0]
|
||||
x,y = line.coords.xy
|
||||
ysup = y[-1] - y[0] if y[0] < y[-1] else y[0] - y[-1]
|
||||
xsup = x[-1] - x[0] if x[0] < x[-1] else x[0] - x[-1]
|
||||
x[0] = x[0] + xsup if x[0] > x[-1] else x[0] - xsup
|
||||
x[-1] = x[-1] - xsup if x[0] > x[-1] else x[-1] + xsup
|
||||
y[0] = y[0] + ysup if y[0] > y[-1] else y[0] - ysup
|
||||
y[-1] = y[-1] - ysup if y[0] > y[-1] else y[-1] + ysup
|
||||
return LineString(zip(x,y))
|
||||
|
||||
def test_data_ok(df,envelop):
|
||||
test_data_ok = gpd.GeoDataFrame(
|
||||
df,
|
||||
geometry=gpd.points_from_xy(df['Long autre'],df['Lat autre']),
|
||||
crs=27572
|
||||
).to_crs(epsg=2154)
|
||||
test_enveloppe = test_data_ok.intersects(envelop)
|
||||
is_ok = test_data_ok[test_enveloppe].copy()
|
||||
not_ok = test_data_ok[~test_enveloppe].copy()
|
||||
not_ok['Lat autre'] = None
|
||||
not_ok['Long autre'] = None
|
||||
return is_ok.drop('geometry',axis=1),not_ok.drop('geometry',axis=1)
|
||||
|
||||
|
||||
def get_nearest_values(row, other_gdf, grid, point_column='geometry'):
|
||||
"""Find the nearest point and return the corresponding value from specified value column."""
|
||||
|
||||
# Recréation de la maille
|
||||
lonG = row['Latitude grade']
|
||||
latG = row['Longitude grade']
|
||||
long = grid[grid.Champ1 == lonG].unary_union
|
||||
lat = grid[grid.Champ1 == latG].unary_union
|
||||
longSup = grid[grid.Champ1 == round((lonG+0.0025),4)].unary_union
|
||||
latSup = grid[grid.Champ1 == round((latG+0.0025),4)].unary_union
|
||||
|
||||
if longSup is None:
|
||||
longInf = grid[grid.Champ1 == round((lonG-0.0025),4)].unary_union
|
||||
d = long.distance(longInf)
|
||||
x,y = long.xy
|
||||
y[0] = y[0] + d
|
||||
y[-1] = y[-1] + d
|
||||
longSup = LineString(zip(x,y))
|
||||
|
||||
|
||||
p1 = long.intersection(lat)
|
||||
while p1.is_empty:
|
||||
long = exend_lines(long,lat)
|
||||
lat = exend_lines(lat,long)
|
||||
p1 = long.intersection(lat)
|
||||
|
||||
p2 = longSup.intersection(lat)
|
||||
while p2.is_empty:
|
||||
longSup = exend_lines(longSup,lat)
|
||||
lat = exend_lines(lat,longSup)
|
||||
p2 = longSup.intersection(lat)
|
||||
|
||||
p3 = longSup.intersection(latSup)
|
||||
while p3.is_empty:
|
||||
longSup = exend_lines(longSup,latSup)
|
||||
latSup = exend_lines(latSup,longSup)
|
||||
p3 = longSup.intersection(latSup)
|
||||
|
||||
p4 = long.intersection(latSup)
|
||||
while p4.is_empty:
|
||||
long = exend_lines(long,latSup)
|
||||
latSup = exend_lines(latSup,long)
|
||||
p4 = long.intersection(latSup)
|
||||
|
||||
pointList = [p1, p2, p3, p4, p1]
|
||||
for i,p in enumerate(pointList):
|
||||
if p.geom_type == 'MultiPoint':
|
||||
for pp in list(p.geoms):
|
||||
if p.distance(row.geom) == row.geom.distance(pp):
|
||||
pointList[i] = pp
|
||||
poly = gpd.GeoSeries(Polygon([[p.x, p.y] for p in pointList]).buffer(0),crs=2154)
|
||||
|
||||
# Filtre de l'ensemble fonctionnel
|
||||
other_geom = other_gdf.geometry.name
|
||||
other_gdf_filter = other_gdf[other_gdf.nom_ensemb == row.esmb].copy()
|
||||
|
||||
if other_gdf_filter.contains(row.geom).any():
|
||||
# pass
|
||||
return row.geom
|
||||
elif other_gdf_filter.intersection(poly.unary_union).is_empty.all():
|
||||
pass
|
||||
else:
|
||||
other_gdf_filter[other_geom] = other_gdf_filter.intersection(poly.unary_union)
|
||||
|
||||
# Create an union of the other GeoDataFrame's geometries:
|
||||
other_points = other_gdf_filter[other_geom].unary_union
|
||||
|
||||
# Find the nearest points
|
||||
nearest_geoms = nearest_points(row[point_column], other_points)
|
||||
|
||||
# Get corresponding values from the other df
|
||||
# nearest_data = other_gdf.loc[other_gdf[other_geom] == nearest_geoms[1]]
|
||||
|
||||
# nearest_value = nearest_data[value_column].get_values()[0]
|
||||
|
||||
return nearest_geoms[1]
|
||||
|
||||
def create_grid():
|
||||
|
||||
grid = gpd.read_file(path.join(PATH,'..',FILE_GRID))
|
||||
grid.rename(columns={'Coordonné':'Champ1'},inplace=True)
|
||||
if grid.crs.srs.lower()!='epsg:2154':
|
||||
grid.to_crs(epsg=2154,inplace=True)
|
||||
if grid['Champ1'].dtype == object:
|
||||
grid['Champ1'] = grid['Champ1'].str.replace(',','.').astype(float)
|
||||
grid = grid.dissolve(by='Champ1',as_index=False)
|
||||
|
||||
|
||||
# Filtre Lat/Long
|
||||
long = grid[grid.Champ1 < 25].copy()
|
||||
lat = grid[grid.Champ1 > 25].copy()
|
||||
|
||||
# Identification du point d'intersection
|
||||
grd = gpd.GeoSeries(list(long.unary_union.intersection(lat.unary_union).geoms),crs=2154)\
|
||||
.to_frame()\
|
||||
.rename_geometry('geom')
|
||||
# Récuperation des coordonnées x et y
|
||||
grd['x'] = grd.geom.x.copy()
|
||||
grd['y'] = grd.geom.y.copy()
|
||||
grd = grd.sort_values(['y','x'],ascending=[True,False]).reset_index(drop=True)
|
||||
x0 = (grd.x[1]-grd.x[0])/2
|
||||
y0 = (grd.y[1]-grd.y[0])/2
|
||||
grd = grd.sort_values(['x','y'],ascending=[True,False]).reset_index(drop=True)
|
||||
x1 = (grd.x[1]-grd.x[0])/2
|
||||
y1 = (grd.y[0]-grd.y[1])/2
|
||||
X = x0+x1
|
||||
Y = y0+y1
|
||||
return grid,X,Y
|
||||
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
|
||||
lst_file = listdir(PATH)
|
||||
lst_file.sort()
|
||||
|
||||
for FILE_DATA in lst_file[22:]:
|
||||
|
||||
grid,X,Y = create_grid()
|
||||
envlop = grid.geometry.unary_union.envelope.buffer(10000)
|
||||
data = gpd.pd.read_csv(path.join(PATH,FILE_DATA),sep=";",encoding='Windows 1252') # ['ASCII','Windows 1252']
|
||||
data.dropna(how='all',inplace=True)
|
||||
data.rename(columns=dict_col,inplace=True)
|
||||
|
||||
print('File :',FILE_DATA)
|
||||
# Si Long/Lat est un champ text, transformation des colonnes en nombre.
|
||||
if ('Long autre' in data.columns) :
|
||||
if (data['Long autre'].dtype == object):
|
||||
data['Long autre'] = data['Long autre'].str.replace(',','.').astype(float)
|
||||
if ('Lat autre' in data.columns) :
|
||||
if data['Lat autre'].dtype == object:
|
||||
data['Lat autre'] = data['Lat autre'].str.replace(',','.').astype(float)
|
||||
if data['Longitude grade'].dtype == object:
|
||||
data['Longitude grade'] = data['Longitude grade'].str.replace(',','.').astype(float)
|
||||
if data['Latitude grade'].dtype == object:
|
||||
data['Latitude grade'] = data['Latitude grade'].str.replace(',','.').astype(float)
|
||||
|
||||
data.loc[data['Lat autre']==0,'Lat autre'] = None
|
||||
data.loc[data['Long autre']==0,'Long autre'] = None
|
||||
|
||||
if not ef.is_valid.all():
|
||||
ef.geometry = ef.make_valid()
|
||||
|
||||
# Isolement des données précises
|
||||
data_ok = data.loc[
|
||||
data['Long autre'].notna() & data['Lat autre'].notna()
|
||||
].copy()
|
||||
|
||||
if data.shape[0] == data_ok.shape[0]:
|
||||
continue
|
||||
|
||||
data_ok, data_not_ok = test_data_ok(data_ok,envlop)
|
||||
|
||||
df = gpd.pd.concat([
|
||||
data.loc[
|
||||
data['Long autre'].isna()&data['Lat autre'].isna()
|
||||
].copy(),
|
||||
data_not_ok]
|
||||
).sort_values('Numéro').reset_index() # 'Numero'
|
||||
|
||||
if data.shape[0] == data_ok.shape[0]:
|
||||
continue
|
||||
|
||||
# Data to GéoData
|
||||
gdf_ok = gpd.GeoDataFrame(
|
||||
data_ok,
|
||||
geometry=gpd.points_from_xy(data_ok['Long autre'],data_ok['Lat autre']),
|
||||
crs=27572
|
||||
).to_crs(epsg=2154)
|
||||
|
||||
# Correspondance [ Grades - Grid ]
|
||||
DICT_GRID = dict(zip(grid.Champ1,grid.geometry.to_wkt()))
|
||||
# Replace Grade by L93
|
||||
df[['Lat93','Long93']] = df[['Latitude grade','Longitude grade']].replace(DICT_GRID)
|
||||
|
||||
# Reconstruction du GéoDataframe
|
||||
gdf = gpd.GeoDataFrame(
|
||||
df.copy(),
|
||||
geometry=gpd.GeoSeries.from_wkt(df['Lat93'])\
|
||||
.intersection(gpd.GeoSeries.from_wkt(df['Long93'])),
|
||||
crs=2154
|
||||
)
|
||||
|
||||
geom_empty = gdf.geometry.is_empty
|
||||
if geom_empty.any():
|
||||
for i,row in gdf.loc[geom_empty].iterrows():
|
||||
while wkt.loads(row['Lat93']).intersection(wkt.loads(row['Long93'])).is_empty:
|
||||
row['Lat93'] = exend_lines(
|
||||
wkt.loads(row['Lat93']),
|
||||
wkt.loads(row['Long93'])
|
||||
).wkt
|
||||
row['Long93'] = exend_lines(
|
||||
wkt.loads(row['Long93']),
|
||||
wkt.loads(row['Lat93'])
|
||||
).wkt
|
||||
gdf.loc[i,'geometry'] = wkt.loads(row['Lat93']).intersection(wkt.loads(row['Long93']))
|
||||
|
||||
gdf.rename_geometry('geom',inplace=True)
|
||||
gdf['x'] = gdf.geom.x + X
|
||||
gdf['y'] = gdf.geom.y + Y
|
||||
gdf.set_geometry(gpd.points_from_xy(gdf.x,gdf.y),inplace=True)
|
||||
|
||||
# Test
|
||||
ef_exp = ef.explode(index_parts=True)
|
||||
# [
|
||||
# near(geom,ldt,ldt_union,'id_lieu_dit')
|
||||
# for geom in df.geometrie
|
||||
# ]
|
||||
|
||||
# gdf['Code ensemble fonct']
|
||||
|
||||
gdf['esmb'] = remove_special_char(
|
||||
gdf['Ensemble fonctionnel'].replace(dict_ef)
|
||||
)
|
||||
ef_exp.nom_ensemb = remove_special_char(ef_exp.nom_ensemb)
|
||||
|
||||
gdf['nearest_point'] = gdf.apply(get_nearest_values, other_gdf=ef_exp, grid=grid, point_column="geom", axis=1)
|
||||
gdf.drop(['geom','esmb'],axis=1,inplace=True)
|
||||
gdf.set_geometry('nearest_point',crs=2154,inplace=True)
|
||||
gdf.rename_geometry('geometry',inplace=True)
|
||||
|
||||
if not gdf_ok.empty:
|
||||
gdf = gpd.pd.concat([gdf,gdf_ok])
|
||||
|
||||
FILE_OUT = FILE_DATA.replace('.csv','_reprojected.csv')
|
||||
gdf.to_csv(path.join(PATH,'..','RESULT',FILE_OUT),sep=';',encoding='Windows 1252',index=False)
|
||||
|
||||
FILE_OUT = FILE_OUT.replace('-','_').replace(' ','_')
|
||||
FILE_OUT = FILE_OUT.replace('(','').replace(')','')
|
||||
(gdf
|
||||
.to_file(path.join(PATH,'..','RESULT','result_2.gpkg'),driver='GPKG',layer=FILE_OUT.replace('-','_')))
|
||||
@@ -0,0 +1,80 @@
|
||||
from sqlalchemy import create_engine # pour lecture de la bd Géonature
|
||||
from sqlalchemy.engine import URL
|
||||
from pycen import update_to_sql
|
||||
import pandas as pd
|
||||
|
||||
usr = 'postgres'
|
||||
pdw = 'postgres'
|
||||
bdd = 'serenadb'
|
||||
host = '172.17.0.2'
|
||||
|
||||
eng = URL.create('postgresql+psycopg2',username=usr,password=pdw,host=host,database=bdd)
|
||||
conn = create_engine(eng)
|
||||
|
||||
obs = pd.read_sql_table('rnf_obse',con=conn,schema='serenabase')
|
||||
|
||||
|
||||
|
||||
## Foreign keys table : serenabase.rnf_choi
|
||||
id_cols = obs.columns[obs.columns.str.endswith('_choi_id')]
|
||||
|
||||
update_to_sql(obs[['obse_id',*id_cols]].replace(0,None).astype('Int64'),
|
||||
conn,'rnf_obse','serenabase','obse_id')
|
||||
|
||||
with conn.begin() as cnx:
|
||||
cnx.execute('ALTER TABLE serenabase.rnf_obse ADD CONSTRAINT rnf_obse_rnf_choi_sexe_fk FOREIGN KEY (obse_sex_choi_id) REFERENCES serenabase.rnf_choi(choi_id);')
|
||||
with conn.begin() as cnx:
|
||||
cnx.execute('ALTER TABLE serenabase.rnf_obse ADD CONSTRAINT rnf_obse_rnf_choi_stade_fk FOREIGN KEY (obse_stade_choi_id) REFERENCES serenabase.rnf_choi(choi_id);')
|
||||
with conn.begin() as cnx:
|
||||
cnx.execute('ALTER TABLE serenabase.rnf_obse ADD CONSTRAINT rnf_obse_rnf_choi_ageunit_fk FOREIGN KEY (obse_ageunit_choi_id) REFERENCES serenabase.rnf_choi(choi_id);')
|
||||
with conn.begin() as cnx:
|
||||
cnx.execute('ALTER TABLE serenabase.rnf_obse ADD CONSTRAINT rnf_obse_rnf_choi_abond_fk FOREIGN KEY (obse_abond_choi_id) REFERENCES serenabase.rnf_choi(choi_id);')
|
||||
with conn.begin() as cnx:
|
||||
cnx.execute('ALTER TABLE serenabase.rnf_obse ADD CONSTRAINT rnf_obse_rnf_choi_precis_fk FOREIGN KEY (obse_precis_choi_id) REFERENCES serenabase.rnf_choi(choi_id);')
|
||||
with conn.begin() as cnx:
|
||||
cnx.execute('ALTER TABLE serenabase.rnf_obse ADD CONSTRAINT rnf_obse_rnf_choi_soci_fk FOREIGN KEY (obse_soci_choi_id) REFERENCES serenabase.rnf_choi(choi_id);')
|
||||
with conn.begin() as cnx:
|
||||
cnx.execute('ALTER TABLE serenabase.rnf_obse ADD CONSTRAINT rnf_obse_rnf_choi_comp_fk FOREIGN KEY (obse_comp_choi_id) REFERENCES serenabase.rnf_choi(choi_id);')
|
||||
with conn.begin() as cnx:
|
||||
cnx.execute('ALTER TABLE serenabase.rnf_obse ADD CONSTRAINT rnf_obse_rnf_choi_compdir_fk FOREIGN KEY (obse_compdir_choi_id) REFERENCES serenabase.rnf_choi(choi_id);')
|
||||
with conn.begin() as cnx:
|
||||
cnx.execute('ALTER TABLE serenabase.rnf_obse ADD CONSTRAINT rnf_obse_rnf_choi_contact_fk FOREIGN KEY (obse_contact_choi_id) REFERENCES serenabase.rnf_choi(choi_id);')
|
||||
with conn.begin() as cnx:
|
||||
cnx.execute('ALTER TABLE serenabase.rnf_obse ADD CONSTRAINT rnf_obse_rnf_choi_contact2_fk FOREIGN KEY (obse_contact2_choi_id) REFERENCES serenabase.rnf_choi(choi_id);')
|
||||
with conn.begin() as cnx:
|
||||
cnx.execute('ALTER TABLE serenabase.rnf_obse ADD CONSTRAINT rnf_obse_rnf_choi_activ_fk FOREIGN KEY (obse_activ_choi_id) REFERENCES serenabase.rnf_choi(choi_id);')
|
||||
with conn.begin() as cnx:
|
||||
cnx.execute('ALTER TABLE serenabase.rnf_obse ADD CONSTRAINT rnf_obse_rnf_choi_caract_fk FOREIGN KEY (obse_caract_choi_id) REFERENCES serenabase.rnf_choi(choi_id);')
|
||||
with conn.begin() as cnx:
|
||||
cnx.execute('ALTER TABLE serenabase.rnf_obse ADD CONSTRAINT rnf_obse_rnf_choi_etat_fk FOREIGN KEY (obse_etat_choi_id) REFERENCES serenabase.rnf_choi(choi_id);')
|
||||
with conn.begin() as cnx:
|
||||
cnx.execute('ALTER TABLE serenabase.rnf_obse ADD CONSTRAINT rnf_obse_rnf_choi_derango_fk FOREIGN KEY (obse_derango_choi_id) REFERENCES serenabase.rnf_choi(choi_id);')
|
||||
with conn.begin() as cnx:
|
||||
cnx.execute('ALTER TABLE serenabase.rnf_obse ADD CONSTRAINT rnf_obse_rnf_choi_derangi_fk FOREIGN KEY (obse_derangi_choi_id) REFERENCES serenabase.rnf_choi(choi_id);')
|
||||
with conn.begin() as cnx:
|
||||
cnx.execute('ALTER TABLE serenabase.rnf_obse ADD CONSTRAINT rnf_obse_rnf_choi_nebul_fk FOREIGN KEY (obse_nebul_choi_id) REFERENCES serenabase.rnf_choi(choi_id);')
|
||||
with conn.begin() as cnx:
|
||||
cnx.execute('ALTER TABLE serenabase.rnf_obse ADD CONSTRAINT rnf_obse_rnf_choi_precip_fk FOREIGN KEY (obse_precip_choi_id) REFERENCES serenabase.rnf_choi(choi_id);')
|
||||
with conn.begin() as cnx:
|
||||
cnx.execute('ALTER TABLE serenabase.rnf_obse ADD CONSTRAINT rnf_obse_rnf_choi_vent_fk FOREIGN KEY (obse_vent_choi_id) REFERENCES serenabase.rnf_choi(choi_id);')
|
||||
with conn.begin() as cnx:
|
||||
cnx.execute('ALTER TABLE serenabase.rnf_obse ADD CONSTRAINT rnf_obse_rnf_choi_ventdir_fk FOREIGN KEY (obse_ventdir_choi_id) REFERENCES serenabase.rnf_choi(choi_id);')
|
||||
with conn.begin() as cnx:
|
||||
cnx.execute('ALTER TABLE serenabase.rnf_obse ADD CONSTRAINT rnf_obse_rnf_choi_visib_fk FOREIGN KEY (obse_visib_choi_id) REFERENCES serenabase.rnf_choi(choi_id);')
|
||||
with conn.begin() as cnx:
|
||||
cnx.execute('ALTER TABLE serenabase.rnf_obse ADD CONSTRAINT rnf_obse_rnf_choi_sol_fk FOREIGN KEY (obse_sol_choi_id) REFERENCES serenabase.rnf_choi(choi_id);')
|
||||
with conn.begin() as cnx:
|
||||
cnx.execute('ALTER TABLE serenabase.rnf_obse ADD CONSTRAINT rnf_obse_rnf_choi_eau_fk FOREIGN KEY (obse_eau_choi_id) REFERENCES serenabase.rnf_choi(choi_id);')
|
||||
with conn.begin() as cnx:
|
||||
cnx.execute('ALTER TABLE serenabase.rnf_obse ADD CONSTRAINT rnf_obse_rnf_choi_eaudir_fk FOREIGN KEY (obse_eaudir_choi_id) REFERENCES serenabase.rnf_choi(choi_id);')
|
||||
|
||||
|
||||
id_cols = obs.columns[obs.columns.str.endswith('user_id')]
|
||||
with conn.begin() as cnx:
|
||||
cnx.execute('ALTER TABLE serenabase.rnf_obse ADD CONSTRAINT rnf_obse_rnf_user_crea_fk FOREIGN KEY (obse_crea_user_id) REFERENCES serenabase.rnf_user(user_id);')
|
||||
|
||||
# DETAIL: La clé (obse_lmod_user_id)=(bbb76b84-493a-4d26-8f01-0440d377198b) n'est pas présente dans la table « rnf_user ».
|
||||
# with conn.begin() as cnx:
|
||||
# cnx.execute('ALTER TABLE serenabase.rnf_obse ADD CONSTRAINT rnf_obse_rnf_user_lmod_fk FOREIGN KEY (obse_lmod_user_id) REFERENCES serenabase.rnf_user(user_id);')
|
||||
|
||||
id_cols = obs.columns[(obs.columns.str.endswith('_id'))& (~obs.columns.str.contains('choi'))]
|
||||
@@ -0,0 +1,208 @@
|
||||
from pycen import con_sicen, update_to_sql
|
||||
from sqlalchemy import create_engine # pour lecture de la bd Géonature
|
||||
from sqlalchemy.engine import URL
|
||||
from shapely.geometry import Polygon
|
||||
import geopandas as gpd
|
||||
import pandas as pd
|
||||
import json
|
||||
import matplotlib.pyplot as plt
|
||||
|
||||
PATH = '/home/colas/Documents/9_PROJETS/6_GEONATURE/MIGRATION/RNGL/DUMP/'
|
||||
FILE = 'export_rnngl.txt'
|
||||
|
||||
usr = 'postgres'
|
||||
pdw = 'postgres'
|
||||
bdd = 'serenadb'
|
||||
host = '172.17.0.2'
|
||||
|
||||
eng = URL.create('postgresql+psycopg2',username=usr,password=pdw,host=host,database=bdd)
|
||||
conn = create_engine(eng)
|
||||
|
||||
tax = pd.read_sql_table('rnf_taxo',con=conn,schema='serenarefe')
|
||||
obs = pd.read_sql_table('rnf_obse',con=conn,schema='serenabase')
|
||||
sit = pd.read_sql_table('rnf_site',con=conn,schema='serenabase')
|
||||
exp = pd.read_table(PATH+FILE,sep='\t',encoding='Windows-1252',dtype={'OBSE_NOMBRE':str,'OBSE_ABOND_CHOI':str,'OBSE_DATE':str})
|
||||
exp.columns = exp.columns.str.lower()
|
||||
exp[['obse_lat','obse_lon','obse_nombre','oglat','oglon']] = exp[['obse_lat','obse_lon','obse_nombre','oglat','oglon']].replace(',','.',regex=True)
|
||||
cols_coord = exp.columns[exp.columns.str.endswith('lat|lon')]
|
||||
year_only = exp.obse_date.str.len() == 4
|
||||
month_only = exp.obse_date.str.len() == 6
|
||||
exp.loc[year_only,'obse_date'] = exp.loc[year_only,'obse_date'] + '0101'
|
||||
exp.loc[month_only,'obse_date'] = exp.loc[month_only,'obse_date'] + '01'
|
||||
exp = (gpd.GeoDataFrame(
|
||||
exp,crs=4326,
|
||||
geometry=gpd.points_from_xy(exp.oglon,exp.oglat))
|
||||
.to_crs(2154))
|
||||
exp['obse_date'] = pd.to_datetime(exp.obse_date)
|
||||
|
||||
# Re-création des géométries des sites
|
||||
sit['poly'] = (sit
|
||||
.site_poly.str[9:]
|
||||
.str.split(','))
|
||||
|
||||
sit.loc[sit.poly.notna(),'geom'] = (sit.loc[sit.poly.notna(),'poly']
|
||||
.apply(lambda x: [xx.split(' ') for xx in x if xx])
|
||||
.apply(lambda x: [[float(xxx) for xxx in xx] for xx in x ])
|
||||
.apply(lambda x: Polygon(x)))
|
||||
|
||||
sit = (sit
|
||||
.set_geometry('geom',crs=4326)
|
||||
.to_crs(2154))
|
||||
|
||||
# Création des géométries à partir des coordonnées
|
||||
geom_na = obs.geom_obse.isna()
|
||||
idx = obs[geom_na].obse_id.tolist()
|
||||
|
||||
# Test des données de l'export
|
||||
exp[exp.obse_id.isin(idx)]
|
||||
|
||||
|
||||
# Mise à jour des observations-sites
|
||||
obs_sit = (obs[geom_na]
|
||||
.merge(sit[['site_id','geom']],how='left',left_on='obse_site_id',right_on='site_id')
|
||||
.set_geometry('geom',crs=2154))
|
||||
update_to_sql(
|
||||
obs_sit[['obse_id','geom']].rename_geometry('geom_site'),
|
||||
conn,'rnf_obse','serenabase','obse_id')
|
||||
|
||||
#### IDENTIFICATION SERENA-GLEMPS ON SICEN ####
|
||||
|
||||
|
||||
#### IDENTIFICATION SERENA-GLEMPS ON SICEN ####
|
||||
obs = pd.read_sql_table('rnf_obse',con=conn,schema='serenabase')
|
||||
sic = pd.read_sql_table('saisie_observation',con=con_sicen,schema='saisie')
|
||||
vsi = pd.read_sql_table('v_saisie_observation',con=con_sicen,schema='saisie')
|
||||
# vmsi = pd.read_sql_table('vm_synthese_observations',con=con_sicen,schema='saisie')
|
||||
vmsi = gpd.read_postgis('SELECT * FROM saisie.vm_synthese_observations',con_sicen,crs=2154)
|
||||
vmsi['date_obs'] = pd.to_datetime(vmsi.date_obs)
|
||||
test_obsEXP = exp.obsv_libel.str.contains('BIRON|JUTON|MARRON',na=False)
|
||||
exp[test_obsEXP]
|
||||
|
||||
id_obsGL = obs.obse_id.astype(str).tolist()
|
||||
t1_obsVM = ~vmsi.observateurs_v2.str.contains('BIRON',na=False)
|
||||
t2_obsVM = vmsi.rmq_localisation.str.contains('GRAND-LEMPS|GRAND LEMPS',na=False,case=False)
|
||||
t3_obsVM = ~vmsi.id_origine.isin(exp[test_obsEXP].obse_id.astype(str))
|
||||
t4_obsVM = vmsi.date_obs <= '2019-01-01'
|
||||
t5_obsVM = vmsi.etude != 'Import données Pro Faune Isère'
|
||||
t6_obsVM = vmsi.id_origine.isin(id_obsGL)
|
||||
|
||||
GL = vmsi[t2_obsVM&t1_obsVM].copy()
|
||||
GL['lat'] = GL.to_crs(4326).geometry.y.round(10).astype(str)
|
||||
GL['lon'] = GL.to_crs(4326).geometry.x.round(10).astype(str)
|
||||
|
||||
while (GL.lat.str.len()==13).all() == False:
|
||||
GL.loc[GL.lat.str.len()<13,'lat'] = GL[GL.lat.str.len()<13].lat + '0'
|
||||
|
||||
while (GL.lon.str.len()==12).all() == False:
|
||||
GL.loc[GL.lon.str.len()<12,'lon'] = GL[GL.lon.str.len()<12].lon + '0'
|
||||
|
||||
test = GL[['lat','lon']].eq(exp[['oglat','oglon']])
|
||||
test = GL.intersects(exp.unary_union)
|
||||
GL[test]
|
||||
# 18819 Données dans Sicen Qui concerne le Grand-Lemps
|
||||
data_gl = vmsi[t6_obsVM]
|
||||
t7 = ~vmsi.id_obs.isin(data_gl.id_obs)
|
||||
GL = vmsi[t1_obsVM&t2_obsVM&t4_obsVM&t5_obsVM&t7].copy()
|
||||
4259862565
|
||||
45.4259862565
|
||||
|
||||
id_obs = obs.obse_id.astype(str).tolist()
|
||||
vsi[vsi.id_origine.isin(id_obs)].astype({'id_origine':int}).sort_values('id_origine')
|
||||
vsi[vsi.id_origine.isin(id_obs)].date_obs.max()
|
||||
|
||||
loc_glps = vsi.localisation.str.contains('ETANG DU GRAND-LEMPS',na=False)
|
||||
vsi[loc_glps].observateur.unique()
|
||||
loc_glps = vmsi.rmq_localisation.str.contains('ETANG DU GRAND-LEMPS',na=False)
|
||||
loc_avenir = vmsi.observateurs_v2.str.contains('BIRON|JUTON|MARRON',na=False)
|
||||
vmsi[loc_glps&~loc_avenir].observateurs_v2.unique()
|
||||
(vsi[loc_glps].observateur
|
||||
.astype(list))
|
||||
|
||||
|
||||
## By OBSV_LIBEL + DATE
|
||||
rnn = exp.copy()
|
||||
rnn.obsv_libel = (rnn.obsv_libel
|
||||
.replace(r'\(.*?\)','',regex=True)
|
||||
.str.strip()
|
||||
.replace({' - ':' '},regex=True)
|
||||
.replace({' ':' '},regex=True)
|
||||
.replace({' et ':' & '},regex=True)
|
||||
.replace({'&':','},regex=True)
|
||||
.replace({' , ':', '},regex=True)
|
||||
.replace({' , ':', '},regex=True)
|
||||
.apply(lambda x: ', '.join(sorted(x.split(', ')))))
|
||||
|
||||
sieg = vmsi.copy()
|
||||
sieg.observateurs_v2 = (sieg.observateurs_v2
|
||||
.replace(r'\(.*?\)','',regex=True)
|
||||
.str.strip()
|
||||
.replace({' ':' '},regex=True)
|
||||
.replace({' ,':', '},regex=True)
|
||||
.replace({' ,':', '},regex=True)
|
||||
.apply(lambda x: ', '.join(sorted(x.split(', ')))))
|
||||
|
||||
sit.set_index('site_id',inplace=True)
|
||||
|
||||
lst_etude_serena = [
|
||||
'Protocole de suivi AVENIR',
|
||||
'Veille écologique AVENIR',
|
||||
'Anciennes BD AVENIR',
|
||||
'BIBLIOGRAPHIE',
|
||||
'Échange de données - CORA Rhône',
|
||||
'Veille écologique',
|
||||
'Échange de données - CBNA',
|
||||
'Echange de données - GRPLS',
|
||||
# 'Echange de données (partenaires)'
|
||||
]
|
||||
t1 = sieg.observateurs_v2.isin(rnn.obsv_libel)
|
||||
t2 = sieg.date_obs.isin(rnn.obse_date)
|
||||
t3 = sieg.intersects(sit.iloc[2].geom.buffer(0))
|
||||
t4 = sieg.etude.isin(lst_etude_serena)
|
||||
t4 = sieg.etude=='Échange de données - CORA Rhône'
|
||||
sieg[t1&t2&t3&t4] # Données à écarter. 15212 données dans Sicen qui concerne le Grand-Lemps
|
||||
# 10404 dans le Grand-Lemps avant le 01-01-2018
|
||||
|
||||
date_max = sieg[t1&t2&t3&t4].date_obs.max()
|
||||
rnn[rnn.obse_date <= date_max]
|
||||
t5 = sieg.date_obs==date_max
|
||||
sieg[t1&t2&t3&t4&(~t5)]
|
||||
dat_max2 = sieg[t1&t2&t3&t4&(~t5)].date_obs.max()
|
||||
t6 = sieg.date_obs==dat_max2
|
||||
sieg[t1&t2&t3&t4&(~t5)&(~t6)]
|
||||
sieg[t1&t2&t3&t4&t6]
|
||||
|
||||
dict_col = {
|
||||
'observateurs_v2':'obsv_libel',
|
||||
'date_obs':'obse_date',
|
||||
# 'lot_donnee':'lot_donnee',
|
||||
'etude':'relv_nom',
|
||||
'nom_complet':'taxo_latin_c',
|
||||
# 'geom':'geometry',
|
||||
}
|
||||
fusion = pd.concat(
|
||||
[sieg[t1&t2&t3&t4].rename(columns=dict_col)[[*dict_col.values()]],
|
||||
rnn[[*dict_col.values()]]],
|
||||
)
|
||||
|
||||
|
||||
#### Test equivalence [obse_lat - oglat, obse_lon - oglon]
|
||||
# Columns :
|
||||
# obsv_libel : Observateur
|
||||
# obse_date : date d'Observation
|
||||
|
||||
test = exp[exp.obse_lat.notna()]
|
||||
tlat = test.obse_lat.eq(test.oglat)
|
||||
tlon = test.obse_lon.eq(test.oglon)
|
||||
test[~tlat].index.equals(test[~tlon].index)
|
||||
test[~tlat]
|
||||
test[~tlon]
|
||||
|
||||
lst_tax = tax.taxo_id.tolist()
|
||||
test = obs[~obs.obse_taxo_id.isin(lst_tax)]
|
||||
|
||||
# ERREUR: une instruction insert ou update sur la table « rnf_obse » viole la contrainte de clé
|
||||
# étrangère « rnf_obse_rnf_user_fk »
|
||||
# Détail : La clé (obse_obsv_id)=(6052) n'est pas présente dans la table « rnf_user ».
|
||||
test = ~vmsi.observateurs_v2.str.contains('Isère|Personnel',na=False,case=False)
|
||||
vmsi[test]
|
||||
vmsi[test].observateurs_v2
|
||||
@@ -0,0 +1,104 @@
|
||||
from pycen import con_gn
|
||||
import pandas as pd
|
||||
import os
|
||||
|
||||
|
||||
def get_synthese(con,lst):
|
||||
sql = '''
|
||||
SELECT * FROM gn_synthese.v_synthese_for_export
|
||||
WHERE cd_nom IN {lst_cdnom}
|
||||
'''.format(lst_cdnom=tuple(lst))
|
||||
return pd.read_sql_query(sql,con)
|
||||
|
||||
if __name__ == "__main__":
|
||||
PATH = '/media/cgeier/SRV/homer_geonat/home/geonatureadmin/taxhub/tmp'
|
||||
file = 'liste_changements.csv'
|
||||
chgt = pd.read_csv(os.path.join(PATH,file))
|
||||
|
||||
updt_cdnom = chgt.grappe_change=='cas1'
|
||||
cd_change = (chgt[~updt_cdnom].i_cd_nom_list
|
||||
.str.removeprefix('[')
|
||||
.str.removesuffix(']')
|
||||
.str.split(', ')
|
||||
.explode()
|
||||
.unique())
|
||||
df = get_synthese(con_gn,cd_change)
|
||||
|
||||
test = df.cd_nom == 134367
|
||||
df[test]
|
||||
df[test].observateurs
|
||||
df[test].nom_valide
|
||||
df[test].nom_vernaculaire
|
||||
|
||||
######
|
||||
### MODIFICATION A APPORTER DANS LA BDD AVANT EVOLUTION DU REFERENTIEL
|
||||
######
|
||||
|
||||
# 100142 OK - Reste inchangé, synonyme devient sous-espèce
|
||||
# 135069 Devient sous-espèce de 100142 49 Observations - GUEDOU Alix (CENI) & GOURGUES Frédéric (Gentiana) & ...
|
||||
### [Geranium robertianum L., 1753 --> Geranium robertianum subsp. robertianum L., 1753]
|
||||
### Mise à jour des obs 135069 --> 100142 Geranium robertianum L., 1753
|
||||
|
||||
# 126650 OK - Reste inchangé, synonyme 141769 devient sous-espèce
|
||||
# 141769 Devient sous-espèce de 126650 1 Observation 2008/10/30 - Raspail Loïc (AAIP)
|
||||
### [Tilia platyphyllos Scop., 1771 --> Tilia platyphyllos subsp. pseudorubra C.K.Schneid., 1909]
|
||||
### Mise à jour des obs 141769 --> 126650 Tilia platyphyllos
|
||||
### FAIT (modification dans l'import source)
|
||||
|
||||
######
|
||||
### LAISSER SUIVRE SON COURS
|
||||
######
|
||||
|
||||
# FLORE
|
||||
########
|
||||
# 145374 OK - Reste var. CD_REF, synonyme 619051 devient synonyme espèce. 619051 Non observé.
|
||||
# 33901 OK - Reste CD_REF, 33781 devient synonymes. 33781 Non observé.
|
||||
# 465265 OK - Reste synonyme de 966674. 966675 devient 2e synonyme. 966675 Non observé.
|
||||
# 53451 OK - Reste CD_REF, 9 synonymes deviennent espèces. Synonymes Non observé.
|
||||
# 89840 OK - Reste CD_REF, 619051 devient synonymes. 619051 Non observé.
|
||||
|
||||
# 108770 Devient synonyme de 762076 4 Observations - BIRON Nicolas (CENI) & GRANGE Benjamin (CENI)
|
||||
### [Monotropa hypopitys L., 1753 --> Hypopitys monotropa Crantz, 1766]
|
||||
# Vérifier espèce présente isère .. Oui
|
||||
|
||||
# 113620 OK - Reste CD_REF, taxon 138815,923321 deviennent synonymes
|
||||
# 138815 Regroupé avec 113620 11 Observations - BIRON Nicolas (CENI)
|
||||
### [Pinguicula grandiflora subsp. grandiflora Lam., 1789 --> Pinguicula grandiflora Lam., 1789]
|
||||
# Regarder champs remarque si pas tagué rosea .. Non
|
||||
# Vérifier espèce présente isère .. Oui
|
||||
|
||||
# 116759 OK - Reste CD_REF, taxon 139596 devient synonyme
|
||||
# 139596 Regroupé avec 116759 1 Observation 2004/12/01 - GILIOTTI Sébastien (Liparis SG)
|
||||
### [Quercus robur var. robur L., 1753 --> Quercus robur L., 1753]
|
||||
|
||||
# 97508 OK - Reste CD_REF, taxon 134367 devient synonyme
|
||||
# 134367 Regroupé avec 97508 1 Observation 2018/07/11 - Pont Bernard (AAIP)
|
||||
### [Euphorbia esula subsp. esula L., 1753 --> Euphorbia esula L., 1753]
|
||||
### Anomalie - modification dans l'import source
|
||||
|
||||
# FONGE
|
||||
########
|
||||
# 814216 OK - Reste CD_REF, 465268 devient synonymes
|
||||
# 465268 Devient synonyme de 814216 3 Observation 2015/xx/xx - RIVOIRE Bernard (MYCOPOLYDEV)
|
||||
### [Porodaedalea conchata (Pers.) Fiasson & Niemelä, 1984 --> Phellinopsis conchata (Pers.) Y.C.Dai, 2010]
|
||||
|
||||
# 44992 OK - Reste CD_REF, 470268 devient synonymes
|
||||
# 470268 Devient synonyme de 44992 2 Observation 2015/xx/xx - RIVOIRE Bernard (MYCOPOLYDEV)
|
||||
### [Tremella mesenterica f. crystallina Ew.Gerhardt, 1997 --> Tremella mesenterica Retz., 1769]
|
||||
|
||||
|
||||
lstjdd_plt = [
|
||||
'[CNR Restauration] Prospection flore vasculaire',
|
||||
'[ENS méandre des Oves] Observation opportuniste par observation directe',
|
||||
'[MC CNR Inspira] Inventaire dendrométrique + DMH',
|
||||
'[MC CNR Inspira] Relevés phytosociologiques et sondages pédologiques',
|
||||
'[N2000 Platière] Inventaire dendrométrique',
|
||||
'[RNN Platière] Observation opportuniste par observation directe',
|
||||
'[RNN Platière] Inventaire dendrométrique en plein',
|
||||
'[RNN Platière] Suivi phytosciologique grèves alluviales',
|
||||
'[RNN Platière] Prospection flore vasculaire',
|
||||
'[CNPE] Suivi phytosciologique des milieux ouverts',
|
||||
'[CNPE] Observation opportuniste par observation directe',
|
||||
'[ABC St-Maurice] Observation Opportuniste par observation directe'
|
||||
]
|
||||
|
||||
Reference in New Issue
Block a user