iop
/
superset
4
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

feat: new CSV upload form and API (#27840)

This commit is contained in:
Daniel Vaz Gaspar 2024-04-15 09:38:51 +01:00 committed by GitHub
parent 40e77be813
commit 54387b4589
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
30 changed files with 2883 additions and 873 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

419
RESOURCES/STANDARD_ROLES.md
View File

@ -17,213 +17,212 @@ specific language governing permissions and limitations
under the License.
-->
||Admin|Alpha|Gamma|SQL_LAB|
|---|---|---|---|---|
|Permission/role description|Admins have all possible rights, including granting or revoking rights from other users and altering other people’s slices and dashboards.|Alpha users have access to all data sources, but they cannot grant or revoke access from other users. They are also limited to altering the objects that they own. Alpha users can add and alter data sources.|Gamma users have limited access. They can only consume data coming from data sources they have been given access to through another complementary role. They only have access to view the slices and dashboards made from data sources that they have access to. Currently Gamma users are not able to alter or add data sources. We assume that they are mostly content consumers, though they can create slices and dashboards.|The sql_lab role grants access to SQL Lab. Note that while Admin users have access to all databases by default, both Alpha and Gamma users need to be given access on a per database basis.||
|can read on SavedQuery|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|
|can write on SavedQuery|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|
|can read on CssTemplate|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can write on CssTemplate|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can read on ReportSchedule|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can write on ReportSchedule|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can read on Chart|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can write on Chart|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can read on Annotation|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can write on Annotation|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can read on Dataset|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can write on Dataset|:heavy_check_mark:|:heavy_check_mark:|O|O|
|can read on Log|:heavy_check_mark:|O|O|O|
|can write on Log|:heavy_check_mark:|O|O|O|
|can read on Dashboard|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can write on Dashboard|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can read on Database|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|
|can write on Database|:heavy_check_mark:|O|O|O|
|can read on Query|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|
|can this form get on ResetPasswordView|:heavy_check_mark:|O|O|O|
|can this form post on ResetPasswordView|:heavy_check_mark:|O|O|O|
|can this form get on ResetMyPasswordView|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can this form post on ResetMyPasswordView|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can this form get on UserInfoEditView|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can this form post on UserInfoEditView|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can show on UserDBModelView|:heavy_check_mark:|O|O|O|
|can edit on UserDBModelView|:heavy_check_mark:|O|O|O|
|can delete on UserDBModelView|:heavy_check_mark:|O|O|O|
|can add on UserDBModelView|:heavy_check_mark:|O|O|O|
|can list on UserDBModelView|:heavy_check_mark:|O|O|O|
|can userinfo on UserDBModelView|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|resetmypassword on UserDBModelView|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|resetpasswords on UserDBModelView|:heavy_check_mark:|O|O|O|
|userinfoedit on UserDBModelView|:heavy_check_mark:|O|O|O|
|can show on RoleModelView|:heavy_check_mark:|O|O|O|
|can edit on RoleModelView|:heavy_check_mark:|O|O|O|
|can delete on RoleModelView|:heavy_check_mark:|O|O|O|
|can add on RoleModelView|:heavy_check_mark:|O|O|O|
|can list on RoleModelView|:heavy_check_mark:|O|O|O|
|copyrole on RoleModelView|:heavy_check_mark:|O|O|O|
|can get on OpenApi|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can show on SwaggerView|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can get on MenuApi|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can list on AsyncEventsRestApi|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can invalidate on CacheRestApi|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can function names on Database|:heavy_check_mark:|O|O|O|
|can query form data on Api|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can query on Api|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can time range on Api|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can this form get on CsvToDatabaseView|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can this form post on CsvToDatabaseView|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can this form get on ExcelToDatabaseView|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can this form post on ExcelToDatabaseView|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can external metadata on Datasource|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can save on Datasource|:heavy_check_mark:|:heavy_check_mark:|O|O|
|can get on Datasource|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can my queries on SqlLab|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|
|can log on Superset|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can schemas access for csv upload on Superset|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can import dashboards on Superset|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can schemas on Superset|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can sqllab history on Superset|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|
|can publish on Superset|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can csv on Superset|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|
|can slice on Superset|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can sync druid source on Superset|:heavy_check_mark:|O|O|O|
|can explore on Superset|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can approve on Superset|:heavy_check_mark:|O|O|O|
|can explore json on Superset|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can fetch datasource metadata on Superset|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can csrf token on Superset|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can sqllab on Superset|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|
|can select star on Superset|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can warm up cache on Superset|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can sqllab table viz on Superset|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|
|can available domains on Superset|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can request access on Superset|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can dashboard on Superset|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can post on TableSchemaView|:heavy_check_mark:|:heavy_check_mark:|O|O|
|can expanded on TableSchemaView|:heavy_check_mark:|:heavy_check_mark:|O|O|
|can delete on TableSchemaView|:heavy_check_mark:|:heavy_check_mark:|O|O|
|can get on TabStateView|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|
|can post on TabStateView|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|
|can delete query on TabStateView|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|
|can migrate query on TabStateView|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|
|can activate on TabStateView|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|
|can delete on TabStateView|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|
|can put on TabStateView|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|
|can read on SecurityRestApi|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|
|menu access on Security|:heavy_check_mark:|O|O|O|
|menu access on List Users|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|menu access on List Roles|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|menu access on Action Log|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|menu access on Manage|:heavy_check_mark:|:heavy_check_mark:|O|O|
|menu access on Annotation Layers|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|menu access on CSS Templates|:heavy_check_mark:|:heavy_check_mark:|O|O|
|menu access on Import Dashboards|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|menu access on Data|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|menu access on Databases|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|menu access on Datasets|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|menu access on Upload a CSV|:heavy_check_mark:|:heavy_check_mark:|O|O|
|menu access on Upload Excel|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|menu access on Charts|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|menu access on Dashboards|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|menu access on SQL Lab|:heavy_check_mark:|O|O|:heavy_check_mark:|
|menu access on SQL Editor|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|
|menu access on Saved Queries|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|
|menu access on Query Search|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|
|all datasource access on all_datasource_access|:heavy_check_mark:|:heavy_check_mark:|O|O|
|all database access on all_database_access|:heavy_check_mark:|:heavy_check_mark:|O|O|
|all query access on all_query_access|:heavy_check_mark:|O|O|O|
|can edit on UserOAuthModelView|:heavy_check_mark:|O|O|O|
|can list on UserOAuthModelView|:heavy_check_mark:|O|O|O|
|can show on UserOAuthModelView|:heavy_check_mark:|O|O|O|
|can userinfo on UserOAuthModelView|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can add on UserOAuthModelView|:heavy_check_mark:|O|O|O|
|can delete on UserOAuthModelView|:heavy_check_mark:|O|O|O|
|userinfoedit on UserOAuthModelView|:heavy_check_mark:|O|O|O|
|can write on DynamicPlugin|:heavy_check_mark:|O|O|O|
|can edit on DynamicPlugin|:heavy_check_mark:|O|O|O|
|can list on DynamicPlugin|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can show on DynamicPlugin|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can download on DynamicPlugin|:heavy_check_mark:|O|O|O|
|can add on DynamicPlugin|:heavy_check_mark:|O|O|O|
|can delete on DynamicPlugin|:heavy_check_mark:|O|O|O|
|can edit on RowLevelSecurityFiltersModelView|:heavy_check_mark:|O|O|O|
|can list on RowLevelSecurityFiltersModelView|:heavy_check_mark:|O|O|O|
|can show on RowLevelSecurityFiltersModelView|:heavy_check_mark:|O|O|O|
|can download on RowLevelSecurityFiltersModelView|:heavy_check_mark:|O|O|O|
|can add on RowLevelSecurityFiltersModelView|:heavy_check_mark:|O|O|O|
|can delete on RowLevelSecurityFiltersModelView|:heavy_check_mark:|O|O|O|
|muldelete on RowLevelSecurityFiltersModelView|:heavy_check_mark:|O|O|O|
|can external metadata by name on Datasource|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can get value on KV|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can store on KV|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can tagged objects on TagView|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can suggestions on TagView|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can get on TagView|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can post on TagView|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can delete on TagView|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can edit on DashboardEmailScheduleView|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can list on DashboardEmailScheduleView|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can show on DashboardEmailScheduleView|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can add on DashboardEmailScheduleView|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can delete on DashboardEmailScheduleView|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|muldelete on DashboardEmailScheduleView|:heavy_check_mark:|:heavy_check_mark:|O|O|
|can edit on SliceEmailScheduleView|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can list on SliceEmailScheduleView|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can show on SliceEmailScheduleView|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can add on SliceEmailScheduleView|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can delete on SliceEmailScheduleView|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|muldelete on SliceEmailScheduleView|:heavy_check_mark:|:heavy_check_mark:|O|O|
|can edit on AlertModelView|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can list on AlertModelView|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can show on AlertModelView|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can add on AlertModelView|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can delete on AlertModelView|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can list on AlertLogModelView|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can show on AlertLogModelView|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can list on AlertObservationModelView|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can show on AlertObservationModelView|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|menu access on Row Level Security|:heavy_check_mark:|O|O|O|
|menu access on Access requests|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|menu access on Home|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|menu access on Plugins|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|menu access on Dashboard Email Schedules|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|menu access on Chart Emails|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|menu access on Alerts|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|menu access on Alerts & Report|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|menu access on Scan New Datasources|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can share dashboard on Superset|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can share chart on Superset|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can this form get on ColumnarToDatabaseView|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can this form post on ColumnarToDatabaseView|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|menu access on Upload a Columnar file|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can export on Chart|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can write on DashboardFilterStateRestApi|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can read on DashboardFilterStateRestApi|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can write on DashboardPermalinkRestApi|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can read on DashboardPermalinkRestApi|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can delete embedded on Dashboard|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can set embedded on Dashboard|:heavy_check_mark:|O|O|O|
|can export on Dashboard|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can get embedded on Dashboard|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can export on Database|:heavy_check_mark:|O|O|O|
|can export on Dataset|:heavy_check_mark:|:heavy_check_mark:|O|O|
|can write on ExploreFormDataRestApi|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can read on ExploreFormDataRestApi|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can write on ExplorePermalinkRestApi|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can read on ExplorePermalinkRestApi|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can export on ImportExportRestApi|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can import on ImportExportRestApi|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can export on SavedQuery|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|
|can dashboard permalink on Superset|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can grant guest token on SecurityRestApi|:heavy_check_mark:|O|O|O|
|can read on AdvancedDataType|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can read on EmbeddedDashboard|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can duplicate on Dataset|:heavy_check_mark:|:heavy_check_mark:|O|O|
|can read on Explore|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can samples on Datasource|:heavy_check_mark:|:heavy_check_mark:|O|O|
|can read on AvailableDomains|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
|can get or create dataset on Dataset|:heavy_check_mark:|:heavy_check_mark:|O|O|
|can get column values on Datasource|:heavy_check_mark:|:heavy_check_mark:|O|O|
|can export csv on SQLLab|:heavy_check_mark:|O|O|:heavy_check_mark:|
|can get results on SQLLab|:heavy_check_mark:|O|O|:heavy_check_mark:|
|can execute sql query on SQLLab|:heavy_check_mark:|O|O|:heavy_check_mark:|
|can recent activity on Log|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| |Admin|Alpha|Gamma|SQL_LAB|
|--------------------------------------------------|---|---|---|---|
| Permission/role description |Admins have all possible rights, including granting or revoking rights from other users and altering other people’s slices and dashboards.|Alpha users have access to all data sources, but they cannot grant or revoke access from other users. They are also limited to altering the objects that they own. Alpha users can add and alter data sources.|Gamma users have limited access. They can only consume data coming from data sources they have been given access to through another complementary role. They only have access to view the slices and dashboards made from data sources that they have access to. Currently Gamma users are not able to alter or add data sources. We assume that they are mostly content consumers, though they can create slices and dashboards.|The sql_lab role grants access to SQL Lab. Note that while Admin users have access to all databases by default, both Alpha and Gamma users need to be given access on a per database basis.||
| can read on SavedQuery |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|
| can write on SavedQuery |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|
| can read on CssTemplate |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can write on CssTemplate |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can read on ReportSchedule |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can write on ReportSchedule |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can read on Chart |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can write on Chart |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can read on Annotation |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can write on Annotation |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can read on Dataset |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can write on Dataset |:heavy_check_mark:|:heavy_check_mark:|O|O|
| can read on Log |:heavy_check_mark:|O|O|O|
| can write on Log |:heavy_check_mark:|O|O|O|
| can read on Dashboard |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can write on Dashboard |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can read on Database |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|
| can write on Database |:heavy_check_mark:|O|O|O|
| can read on Query |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|
| can this form get on ResetPasswordView |:heavy_check_mark:|O|O|O|
| can this form post on ResetPasswordView |:heavy_check_mark:|O|O|O|
| can this form get on ResetMyPasswordView |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can this form post on ResetMyPasswordView |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can this form get on UserInfoEditView |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can this form post on UserInfoEditView |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can show on UserDBModelView |:heavy_check_mark:|O|O|O|
| can edit on UserDBModelView |:heavy_check_mark:|O|O|O|
| can delete on UserDBModelView |:heavy_check_mark:|O|O|O|
| can add on UserDBModelView |:heavy_check_mark:|O|O|O|
| can list on UserDBModelView |:heavy_check_mark:|O|O|O|
| can userinfo on UserDBModelView |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| resetmypassword on UserDBModelView |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| resetpasswords on UserDBModelView |:heavy_check_mark:|O|O|O|
| userinfoedit on UserDBModelView |:heavy_check_mark:|O|O|O|
| can show on RoleModelView |:heavy_check_mark:|O|O|O|
| can edit on RoleModelView |:heavy_check_mark:|O|O|O|
| can delete on RoleModelView |:heavy_check_mark:|O|O|O|
| can add on RoleModelView |:heavy_check_mark:|O|O|O|
| can list on RoleModelView |:heavy_check_mark:|O|O|O|
| copyrole on RoleModelView |:heavy_check_mark:|O|O|O|
| can get on OpenApi |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can show on SwaggerView |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can get on MenuApi |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can list on AsyncEventsRestApi |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can invalidate on CacheRestApi |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can function names on Database |:heavy_check_mark:|O|O|O|
| can csv upload on Database |:heavy_check_mark:|O|O|O|
| can query form data on Api |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can query on Api |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can time range on Api |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can this form get on ExcelToDatabaseView |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can this form post on ExcelToDatabaseView |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can external metadata on Datasource |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can save on Datasource |:heavy_check_mark:|:heavy_check_mark:|O|O|
| can get on Datasource |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can my queries on SqlLab |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|
| can log on Superset |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can schemas access for csv upload on Superset |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can import dashboards on Superset |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can schemas on Superset |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can sqllab history on Superset |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|
| can publish on Superset |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can csv on Superset |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|
| can slice on Superset |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can sync druid source on Superset |:heavy_check_mark:|O|O|O|
| can explore on Superset |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can approve on Superset |:heavy_check_mark:|O|O|O|
| can explore json on Superset |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can fetch datasource metadata on Superset |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can csrf token on Superset |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can sqllab on Superset |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|
| can select star on Superset |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can warm up cache on Superset |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can sqllab table viz on Superset |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|
| can available domains on Superset |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can request access on Superset |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can dashboard on Superset |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can post on TableSchemaView |:heavy_check_mark:|:heavy_check_mark:|O|O|
| can expanded on TableSchemaView |:heavy_check_mark:|:heavy_check_mark:|O|O|
| can delete on TableSchemaView |:heavy_check_mark:|:heavy_check_mark:|O|O|
| can get on TabStateView |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|
| can post on TabStateView |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|
| can delete query on TabStateView |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|
| can migrate query on TabStateView |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|
| can activate on TabStateView |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|
| can delete on TabStateView |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|
| can put on TabStateView |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|
| can read on SecurityRestApi |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|
| menu access on Security |:heavy_check_mark:|O|O|O|
| menu access on List Users |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| menu access on List Roles |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| menu access on Action Log |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| menu access on Manage |:heavy_check_mark:|:heavy_check_mark:|O|O|
| menu access on Annotation Layers |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| menu access on CSS Templates |:heavy_check_mark:|:heavy_check_mark:|O|O|
| menu access on Import Dashboards |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| menu access on Data |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| menu access on Databases |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| menu access on Datasets |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| menu access on Upload a CSV |:heavy_check_mark:|:heavy_check_mark:|O|O|
| menu access on Upload Excel |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| menu access on Charts |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| menu access on Dashboards |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| menu access on SQL Lab |:heavy_check_mark:|O|O|:heavy_check_mark:|
| menu access on SQL Editor |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|
| menu access on Saved Queries |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|
| menu access on Query Search |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|
| all datasource access on all_datasource_access |:heavy_check_mark:|:heavy_check_mark:|O|O|
| all database access on all_database_access |:heavy_check_mark:|:heavy_check_mark:|O|O|
| all query access on all_query_access |:heavy_check_mark:|O|O|O|
| can edit on UserOAuthModelView |:heavy_check_mark:|O|O|O|
| can list on UserOAuthModelView |:heavy_check_mark:|O|O|O|
| can show on UserOAuthModelView |:heavy_check_mark:|O|O|O|
| can userinfo on UserOAuthModelView |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can add on UserOAuthModelView |:heavy_check_mark:|O|O|O|
| can delete on UserOAuthModelView |:heavy_check_mark:|O|O|O|
| userinfoedit on UserOAuthModelView |:heavy_check_mark:|O|O|O|
| can write on DynamicPlugin |:heavy_check_mark:|O|O|O|
| can edit on DynamicPlugin |:heavy_check_mark:|O|O|O|
| can list on DynamicPlugin |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can show on DynamicPlugin |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can download on DynamicPlugin |:heavy_check_mark:|O|O|O|
| can add on DynamicPlugin |:heavy_check_mark:|O|O|O|
| can delete on DynamicPlugin |:heavy_check_mark:|O|O|O|
| can edit on RowLevelSecurityFiltersModelView |:heavy_check_mark:|O|O|O|
| can list on RowLevelSecurityFiltersModelView |:heavy_check_mark:|O|O|O|
| can show on RowLevelSecurityFiltersModelView |:heavy_check_mark:|O|O|O|
| can download on RowLevelSecurityFiltersModelView |:heavy_check_mark:|O|O|O|
| can add on RowLevelSecurityFiltersModelView |:heavy_check_mark:|O|O|O|
| can delete on RowLevelSecurityFiltersModelView |:heavy_check_mark:|O|O|O|
| muldelete on RowLevelSecurityFiltersModelView |:heavy_check_mark:|O|O|O|
| can external metadata by name on Datasource |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can get value on KV |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can store on KV |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can tagged objects on TagView |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can suggestions on TagView |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can get on TagView |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can post on TagView |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can delete on TagView |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can edit on DashboardEmailScheduleView |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can list on DashboardEmailScheduleView |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can show on DashboardEmailScheduleView |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can add on DashboardEmailScheduleView |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can delete on DashboardEmailScheduleView |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| muldelete on DashboardEmailScheduleView |:heavy_check_mark:|:heavy_check_mark:|O|O|
| can edit on SliceEmailScheduleView |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can list on SliceEmailScheduleView |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can show on SliceEmailScheduleView |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can add on SliceEmailScheduleView |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can delete on SliceEmailScheduleView |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| muldelete on SliceEmailScheduleView |:heavy_check_mark:|:heavy_check_mark:|O|O|
| can edit on AlertModelView |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can list on AlertModelView |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can show on AlertModelView |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can add on AlertModelView |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can delete on AlertModelView |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can list on AlertLogModelView |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can show on AlertLogModelView |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can list on AlertObservationModelView |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can show on AlertObservationModelView |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| menu access on Row Level Security |:heavy_check_mark:|O|O|O|
| menu access on Access requests |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| menu access on Home |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| menu access on Plugins |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| menu access on Dashboard Email Schedules |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| menu access on Chart Emails |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| menu access on Alerts |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| menu access on Alerts & Report |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| menu access on Scan New Datasources |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can share dashboard on Superset |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can share chart on Superset |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can this form get on ColumnarToDatabaseView |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can this form post on ColumnarToDatabaseView |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| menu access on Upload a Columnar file |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can export on Chart |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can write on DashboardFilterStateRestApi |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can read on DashboardFilterStateRestApi |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can write on DashboardPermalinkRestApi |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can read on DashboardPermalinkRestApi |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can delete embedded on Dashboard |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can set embedded on Dashboard |:heavy_check_mark:|O|O|O|
| can export on Dashboard |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can get embedded on Dashboard |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can export on Database |:heavy_check_mark:|O|O|O|
| can export on Dataset |:heavy_check_mark:|:heavy_check_mark:|O|O|
| can write on ExploreFormDataRestApi |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can read on ExploreFormDataRestApi |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can write on ExplorePermalinkRestApi |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can read on ExplorePermalinkRestApi |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can export on ImportExportRestApi |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can import on ImportExportRestApi |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can export on SavedQuery |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|
| can dashboard permalink on Superset |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can grant guest token on SecurityRestApi |:heavy_check_mark:|O|O|O|
| can read on AdvancedDataType |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can read on EmbeddedDashboard |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can duplicate on Dataset |:heavy_check_mark:|:heavy_check_mark:|O|O|
| can read on Explore |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can samples on Datasource |:heavy_check_mark:|:heavy_check_mark:|O|O|
| can read on AvailableDomains |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can get or create dataset on Dataset |:heavy_check_mark:|:heavy_check_mark:|O|O|
| can get column values on Datasource |:heavy_check_mark:|:heavy_check_mark:|O|O|
| can export csv on SQLLab |:heavy_check_mark:|O|O|:heavy_check_mark:|
| can get results on SQLLab |:heavy_check_mark:|O|O|:heavy_check_mark:|
| can execute sql query on SQLLab |:heavy_check_mark:|O|O|:heavy_check_mark:|
| can recent activity on Log |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|

355
superset-frontend/src/features/databases/CSVUploadModal/CSVUploadModal.test.tsx Normal file
View File

@ -0,0 +1,355 @@
/**
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
import React from 'react';
import fetchMock from 'fetch-mock';
import CSVUploadModal, {
validateUploadFileExtension,
} from 'src/features/databases/CSVUploadModal';
import { render, screen } from 'spec/helpers/testing-library';
import userEvent from '@testing-library/user-event';
import { waitFor } from '@testing-library/react';
import { UploadFile } from 'antd/lib/upload/interface';
import { forEach } from 'lodash';
fetchMock.post('glob:*api/v1/database/1/csv_upload/', {});
fetchMock.get(
'glob:*api/v1/database/?q=(filters:!((col:allow_file_upload,opr:eq,value:!t)),page:0,page_size:100)',
{
result: [
{
id: 1,
database_name: 'database1',
},
{
id: 2,
database_name: 'database2',
},
],
},
);
fetchMock.get('glob:*api/v1/database/1/schemas/', {
result: ['information_schema', 'public'],
});
fetchMock.get('glob:*api/v1/database/2/schemas/', {
result: ['schema1', 'schema2'],
});
const csvProps = {
show: true,
onHide: () => {},
allowedExtensions: ['csv', 'tsv'],
};
test('renders the general information elements correctly', () => {
render(<CSVUploadModal {...csvProps} />, {
useRedux: true,
});
const cancelButton = screen.getByRole('button', {
name: 'Cancel',
});
const uploadButton = screen.getByRole('button', {
name: 'Upload',
});
const selectButton = screen.getByRole('button', {
name: 'Select',
});
const title = screen.getByRole('heading', {
name: /csv upload/i,
});
const panel1 = screen.getByRole('heading', {
name: /General information/i,
});
const panel2 = screen.getByRole('heading', {
name: /file settings/i,
});
const panel3 = screen.getByRole('heading', {
name: /columns/i,
});
const panel4 = screen.getByRole('heading', {
name: /rows/i,
});
const selectDatabase = screen.getByRole('combobox', {
name: /select a database/i,
});
const selectDelimiter = screen.getByRole('combobox', {
name: /choose a delimiter/i,
});
const inputTableName = screen.getByRole('textbox', {
name: /table name/i,
});
const inputSchema = screen.getByRole('combobox', {
name: /schema/i,
});
const visibleComponents = [
cancelButton,
uploadButton,
selectButton,
title,
panel1,
panel2,
panel3,
panel4,
selectDatabase,
selectDelimiter,
inputTableName,
inputSchema,
];
visibleComponents.forEach(component => {
expect(component).toBeVisible();
});
});
test('renders the file settings elements correctly', () => {
render(<CSVUploadModal {...csvProps} />, {
useRedux: true,
});
expect(screen.queryByText('If Table Already Exists')).not.toBeInTheDocument();
const panelHeader = screen.getByRole('heading', {
name: /file settings/i,
});
userEvent.click(panelHeader);
const selectTableAlreadyExists = screen.getByRole('combobox', {
name: /choose already exists/i,
});
const switchSkipInitialSpace = screen.getByTestId('skipInitialSpace');
const switchSkipBlankLines = screen.getByTestId('skipBlankLines');
const switchDayFirst = screen.getByTestId('dayFirst');
const inputDecimalCharacter = screen.getByRole('textbox', {
name: /decimal character/i,
});
const selectColumnsDates = screen.getByRole('combobox', {
name: /choose columns to be parsed as dates/i,
});
const selectNullValues = screen.getByRole('combobox', {
name: /null values/i,
});
userEvent.click(selectColumnsDates);
userEvent.click(selectNullValues);
const visibleComponents = [
selectTableAlreadyExists,
switchSkipInitialSpace,
switchDayFirst,
switchSkipBlankLines,
inputDecimalCharacter,
selectNullValues,
];
visibleComponents.forEach(component => {
expect(component).toBeVisible();
});
});
test('renders the columns elements correctly', () => {
render(<CSVUploadModal {...csvProps} />, {
useRedux: true,
});
const panelHeader = screen.getByRole('heading', {
name: /columns/i,
});
userEvent.click(panelHeader);
const selectIndexColumn = screen.getByRole('combobox', {
name: /Choose index column/i,
});
const switchDataFrameIndex = screen.getByTestId('dataFrameIndex');
const inputColumnLabels = screen.getByRole('textbox', {
name: /Column labels/i,
});
const selectColumnsToRead = screen.getByRole('combobox', {
name: /Choose columns to read/i,
});
const switchOverwriteDuplicates = screen.getByTestId('overwriteDuplicates');
const inputColumnDataTypes = screen.getByRole('textbox', {
name: /Column data types/i,
});
userEvent.click(selectColumnsToRead);
const visibleComponents = [
selectIndexColumn,
switchDataFrameIndex,
inputColumnLabels,
selectColumnsToRead,
switchOverwriteDuplicates,
inputColumnDataTypes,
];
visibleComponents.forEach(component => {
expect(component).toBeVisible();
});
});
test('renders the rows elements correctly', () => {
render(<CSVUploadModal {...csvProps} />, {
useRedux: true,
});
const panelHeader = screen.getByRole('heading', {
name: /rows/i,
});
userEvent.click(panelHeader);
const inputHeaderRow = screen.getByRole('spinbutton', {
name: /header row/i,
});
const inputRowsToRead = screen.getByRole('spinbutton', {
name: /rows to read/i,
});
const inputSkipRows = screen.getByRole('spinbutton', {
name: /skip rows/i,
});
const visibleComponents = [inputHeaderRow, inputRowsToRead, inputSkipRows];
visibleComponents.forEach(component => {
expect(component).toBeVisible();
});
});
test('database and schema are correctly populated', async () => {
render(<CSVUploadModal {...csvProps} />, {
useRedux: true,
});
const selectDatabase = screen.getByRole('combobox', {
name: /select a database/i,
});
const selectSchema = screen.getByRole('combobox', {
name: /schema/i,
});
userEvent.click(selectDatabase);
await waitFor(() => screen.getByText('database1'));
await waitFor(() => screen.getByText('database2'));
screen.getByText('database1').click();
userEvent.click(selectSchema);
// make sure the schemas for database1 are displayed
await waitFor(() => screen.getAllByText('information_schema'));
await waitFor(() => screen.getAllByText('public'));
screen.getByText('database2').click();
userEvent.click(selectSchema);
// make sure the schemas for database2 are displayed
await waitFor(() => screen.getAllByText('schema1'));
await waitFor(() => screen.getAllByText('schema2'));
});
test('form without required fields', async () => {
render(<CSVUploadModal {...csvProps} />, {
useRedux: true,
});
const uploadButton = screen.getByRole('button', {
name: 'Upload',
});
// Submit form without filling any fields
userEvent.click(uploadButton);
await waitFor(() => screen.getByText('Uploading a file is required'));
await waitFor(() => screen.getByText('Selecting a database is required'));
await waitFor(() => screen.getByText('Table name is required'));
});
test('form post', async () => {
render(<CSVUploadModal {...csvProps} />, {
useRedux: true,
});
const selectButton = screen.getByRole('button', {
name: 'Select',
});
userEvent.click(selectButton);
// Select a file from the file dialog
const file = new File(['test'], 'test.csv', { type: 'text' });
const inputElement = document.querySelector('input[type="file"]');
if (inputElement) {
userEvent.upload(inputElement, file);
}
const selectDatabase = screen.getByRole('combobox', {
name: /select a database/i,
});
userEvent.click(selectDatabase);
await waitFor(() => screen.getByText('database1'));
await waitFor(() => screen.getByText('database2'));
screen.getByText('database1').click();
const selectSchema = screen.getByRole('combobox', {
name: /schema/i,
});
userEvent.click(selectSchema);
await waitFor(() => screen.getAllByText('public'));
screen.getAllByText('public')[1].click();
// Fill out form fields
const inputTableName = screen.getByRole('textbox', {
name: /table name/i,
});
userEvent.type(inputTableName, 'table1');
const uploadButton = screen.getByRole('button', {
name: 'Upload',
});
userEvent.click(uploadButton);
await waitFor(() => fetchMock.called('glob:*api/v1/database/1/csv_upload/'));
// Get the matching fetch calls made
const matchingCalls = fetchMock.calls('glob:*api/v1/database/1/csv_upload/');
expect(matchingCalls).toHaveLength(1);
const [_, options] = matchingCalls[0];
const formData = options?.body as FormData;
expect(formData.get('table_name')).toBe('table1');
expect(formData.get('schema')).toBe('public');
expect(formData.get('table_name')).toBe('table1');
const fileData = formData.get('file') as File;
expect(fileData.name).toBe('test.csv');
});
test('validate file extension returns false', () => {
const invalidFileNames = ['out', 'out.exe', 'out.csv.exe', '.csv'];
forEach(invalidFileNames, fileName => {
const file: UploadFile<any> = {
name: fileName,
uid: 'xp',
size: 100,
type: 'text/csv',
};
expect(validateUploadFileExtension(file, ['csv', 'tsv'])).toBe(false);
});
});
test('validate file extension returns true', () => {
const invalidFileNames = ['out.csv', 'out.tsv', 'out.exe.csv', 'out a.csv'];
forEach(invalidFileNames, fileName => {
const file: UploadFile<any> = {
name: fileName,
uid: 'xp',
size: 100,
type: 'text/csv',
};
expect(validateUploadFileExtension(file, ['csv', 'tsv'])).toBe(true);
});
});

54
superset-frontend/src/features/databases/CSVUploadModal/ColumnsPreview.tsx Normal file
View File

@ -0,0 +1,54 @@
/**
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
import React from 'react';
import { styled, t } from '@superset-ui/core';
import { Typography } from 'src/components';
import TagsList from 'src/components/Tags/TagsList';
import TagType from 'src/types/TagType';
interface ColumnsPreviewProps {
columns: string[];
maxColumnsToShow?: number;
}
export const StyledDivContainer = styled.div`
//margin-top: 10px;
//margin-bottom: 10px;
`;
const ColumnsPreview: React.FC<ColumnsPreviewProps> = ({
columns,
maxColumnsToShow = 4,
}) => {
const tags: TagType[] = columns.map(column => ({ name: column }));
return (
<StyledDivContainer>
<Typography.Text type="secondary">Columns:</Typography.Text>
{columns.length === 0 ? (
<p className="help-block">{t('Upload CSV file to preview columns')}</p>
) : (
<TagsList tags={tags} maxTags={maxColumnsToShow} />
)}
</StyledDivContainer>
);
};
export default ColumnsPreview;

53
superset-frontend/src/features/databases/CSVUploadModal/StyledFormItemWithTip.tsx Normal file
View File

@ -0,0 +1,53 @@
/**
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
import React from 'react';
import InfoTooltip from 'src/components/InfoTooltip';
import { StyledFormItem } from './styles';
interface StyledFormItemWithTipProps {
label: string;
tip: string;
name: string;
children: React.ReactNode;
rules?: any[];
}
const StyledFormItemWithTip: React.FC<StyledFormItemWithTipProps> = ({
label,
tip,
children,
name,
rules,
}) => (
<StyledFormItem
label={
<div>
{label}
<InfoTooltip tooltip={tip} />
</div>
}
name={name}
rules={rules}
>
{children}
</StyledFormItem>
);
export default StyledFormItemWithTip;

841
superset-frontend/src/features/databases/CSVUploadModal/index.tsx Normal file
View File

@ -0,0 +1,841 @@
/**
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
import React, { FunctionComponent, useEffect, useMemo, useState } from 'react';
import {
getClientErrorObject,
SupersetClient,
SupersetTheme,
t,
} from '@superset-ui/core';
import Modal from 'src/components/Modal';
import Button from 'src/components/Button';
import { Switch } from 'src/components/Switch';
import Collapse from 'src/components/Collapse';
import {
Upload,
AntdForm,
Col,
Row,
AsyncSelect,
Select,
} from 'src/components';
import { UploadOutlined } from '@ant-design/icons';
import { Input, InputNumber } from 'src/components/Input';
import rison from 'rison';
import { UploadChangeParam, UploadFile } from 'antd/lib/upload/interface';
import withToasts from 'src/components/MessageToasts/withToasts';
import {
antDModalStyles,
antDModalNoPaddingStyles,
antdCollapseStyles,
formStyles,
StyledFormItem,
StyledSwitchContainer,
} from './styles';
import ColumnsPreview from './ColumnsPreview';
import StyledFormItemWithTip from './StyledFormItemWithTip';
interface CSVUploadModalProps {
addDangerToast: (msg: string) => void;
addSuccessToast: (msg: string) => void;
onHide: () => void;
show: boolean;
allowedExtensions: string[];
}
interface UploadInfo {
database_id: number;
table_name: string;
schema: string;
delimiter: string;
already_exists: string;
skip_initial_space: boolean;
skip_blank_lines: boolean;
day_first: boolean;
decimal_character: string;
null_values: Array<string>;
header_row: string;
rows_to_read: string | null;
skip_rows: string;
column_dates: Array<string>;
index_column: string | null;
dataframe_index: boolean;
column_labels: string;
columns_read: Array<string>;
overwrite_duplicates: boolean;
column_data_types: string;
}
const defaultUploadInfo: UploadInfo = {
database_id: 0,
table_name: '',
schema: '',
delimiter: ',',
already_exists: 'fail',
skip_initial_space: false,
skip_blank_lines: false,
day_first: false,
decimal_character: '.',
null_values: [],
header_row: '0',
rows_to_read: null,
skip_rows: '0',
column_dates: [],
index_column: null,
dataframe_index: false,
column_labels: '',
columns_read: [],
overwrite_duplicates: false,
column_data_types: '',
};
// Allowed extensions to accept for file upload, users can always override this
// by selecting all file extensions on the OS file picker. Also ".txt" will
// allow all files to be selected.
const allowedExtensionsToAccept = '.csv, .tsv';
const READ_HEADER_SIZE = 10000;
export const validateUploadFileExtension = (
file: UploadFile<any>,
allowedExtensions: string[],
) => {
const extensionMatch = file.name.match(/.+\.([^.]+)$/);
if (!extensionMatch) {
return false;
}
const fileType = extensionMatch[1];
return allowedExtensions.includes(fileType);
};
const SwitchContainer: React.FC<{ label: string; dataTest: string }> = ({
label,
dataTest,
children,
}) => (
<StyledSwitchContainer>
<Switch data-test={dataTest} />
<div className="switch-label">{label}</div>
{children}
</StyledSwitchContainer>
);
const CSVUploadModal: FunctionComponent<CSVUploadModalProps> = ({
addDangerToast,
addSuccessToast,
onHide,
show,
allowedExtensions,
}) => {
const [form] = AntdForm.useForm();
// Declare states here
const [currentDatabaseId, setCurrentDatabaseId] = useState<number>(0);
const [fileList, setFileList] = useState<UploadFile[]>([]);
const [columns, setColumns] = React.useState<string[]>([]);
const [delimiter, setDelimiter] = useState<string>(',');
const [isLoading, setIsLoading] = useState<boolean>(false);
const [currentSchema, setCurrentSchema] = useState<string | undefined>();
const nullValuesOptions = [
{
value: '""',
label: 'Empty Strings ""',
},
{
value: 'None',
label: 'None',
},
{
value: 'nan',
label: 'nan',
},
{
value: 'null',
label: 'null',
},
{
value: 'N/A',
label: 'N/A',
},
];
const delimiterOptions = [
{
value: ',',
label: 'Comma ","',
},
{
value: ';',
label: 'Semicolon ";"',
},
{
value: '\t',
label: 'Tab "\\t"',
},
{
value: '|',
label: 'Pipe',
},
];
const tableAlreadyExistsOptions = [
{
value: 'fail',
label: 'Fail',
},
{
value: 'replace',
label: 'Replace',
},
{
value: 'append',
label: 'Append',
},
];
const onChangeDatabase = (database: { value: number; label: string }) => {
setCurrentDatabaseId(database?.value);
setCurrentSchema(undefined);
form.setFieldsValue({ schema: undefined });
};
const onChangeSchema = (schema: { value: string; label: string }) => {
setCurrentSchema(schema?.value);
};
const onChangeDelimiter = (value: string) => {
setDelimiter(value);
};
const clearModal = () => {
setFileList([]);
setColumns([]);
setCurrentSchema('');
setCurrentDatabaseId(0);
setIsLoading(false);
form.resetFields();
};
const loadDatabaseOptions = useMemo(
() =>
(input = '', page: number, pageSize: number) => {
const query = rison.encode_uri({
filters: [
{
col: 'allow_file_upload',
opr: 'eq',
value: true,
},
],
page,
page_size: pageSize,
});
return SupersetClient.get({
endpoint: `/api/v1/database/?q=${query}`,
}).then(response => {
const list = response.json.result.map(
(item: { id: number; database_name: string }) => ({
value: item.id,
label: item.database_name,
}),
);
return { data: list, totalCount: response.json.count };
});
},
[],
);
const loadSchemaOptions = useMemo(
() =>
(input = '', page: number, pageSize: number) => {
if (!currentDatabaseId) {
return Promise.resolve({ data: [], totalCount: 0 });
}
return SupersetClient.get({
endpoint: `/api/v1/database/${currentDatabaseId}/schemas/`,
}).then(response => {
const list = response.json.result.map((item: string) => ({
value: item,
label: item,
}));
return { data: list, totalCount: response.json.count };
});
},
[currentDatabaseId],
);
const onClose = () => {
clearModal();
onHide();
};
const onFinish = () => {
const fields = form.getFieldsValue();
fields.database_id = currentDatabaseId;
fields.schema = currentSchema;
const mergedValues = { ...defaultUploadInfo, ...fields };
const formData = new FormData();
const file = fileList[0]?.originFileObj;
if (file) {
formData.append('file', file);
}
formData.append('delimiter', mergedValues.delimiter);
formData.append('table_name', mergedValues.table_name);
formData.append('schema', mergedValues.schema);
formData.append('already_exists', mergedValues.already_exists);
formData.append('skip_initial_space', mergedValues.skip_initial_space);
formData.append('skip_blank_lines', mergedValues.skip_blank_lines);
formData.append('day_first', mergedValues.day_first);
formData.append('decimal_character', mergedValues.decimal_character);
formData.append('null_values', mergedValues.null_values);
formData.append('header_row', mergedValues.header_row);
if (mergedValues.rows_to_read != null) {
formData.append('rows_to_read', mergedValues.rows_to_read);
}
formData.append('skip_rows', mergedValues.skip_rows);
formData.append('column_dates', mergedValues.column_dates);
if (mergedValues.index_column != null) {
formData.append('index_column', mergedValues.index_column);
}
formData.append('dataframe_index', mergedValues.dataframe_index);
formData.append('column_labels', mergedValues.column_labels);
formData.append('columns_read', mergedValues.columns_read);
formData.append('overwrite_duplicates', mergedValues.overwrite_duplicates);
formData.append('column_data_types', mergedValues.column_data_types);
setIsLoading(true);
return SupersetClient.post({
endpoint: `/api/v1/database/${currentDatabaseId}/csv_upload/`,
body: formData,
headers: { Accept: 'application/json' },
})
.then(() => {
addSuccessToast(t('CSV Imported'));
setIsLoading(false);
onClose();
})
.catch(response =>
getClientErrorObject(response).then(error => {
addDangerToast(error.error || 'Error');
}),
)
.finally(() => {
setIsLoading(false);
});
};
const onRemoveFile = (removedFile: UploadFile) => {
setFileList(fileList.filter(file => file.uid !== removedFile.uid));
setColumns([]);
return false;
};
const columnsToOptions = () =>
columns.map(column => ({
value: column,
label: column,
}));
const readFileContent = (file: File) =>
new Promise<string>((resolve, reject) => {
const reader = new FileReader();
reader.onload = event => {
if (event.target) {
const text = event.target.result as string;
resolve(text);
} else {
reject(new Error('Failed to read file content'));
}
};
reader.onerror = () => {
reject(new Error('Failed to read file content'));
};
reader.readAsText(file.slice(0, READ_HEADER_SIZE));
});
const processFileContent = async (file: File) => {
try {
const text = await readFileContent(file);
const firstLine = text.split('\n')[0].trim();
const firstRow = firstLine
.split(delimiter)
.map(column => column.replace(/^"(.*)"$/, '$1'));
setColumns(firstRow);
} catch (error) {
addDangerToast('Failed to process file content');
}
};
const onChangeFile = async (info: UploadChangeParam<any>) => {
setFileList([
{
...info.file,
status: 'done',
},
]);
await processFileContent(info.file.originFileObj);
};
useEffect(() => {
if (
columns.length > 0 &&
fileList[0].originFileObj &&
fileList[0].originFileObj instanceof File
) {
processFileContent(fileList[0].originFileObj).then(r => r);
}
}, [delimiter]);
const validateUpload = (_: any, value: string) => {
if (fileList.length === 0) {
return Promise.reject(t('Uploading a file is required'));
}
if (!validateUploadFileExtension(fileList[0], allowedExtensions)) {
return Promise.reject(
t(
'Upload a file with a valid extension. Valid: [%s]',
allowedExtensions.join(','),
),
);
}
return Promise.resolve();
};
const validateDatabase = (_: any, value: string) => {
if (!currentDatabaseId) {
return Promise.reject(t('Selecting a database is required'));
}
return Promise.resolve();
};
return (
<Modal
css={(theme: SupersetTheme) => [
antDModalNoPaddingStyles,
antDModalStyles(theme),
formStyles(theme),
]}
primaryButtonLoading={isLoading}
name="database"
data-test="csvupload-modal"
onHandledPrimaryAction={form.submit}
onHide={onClose}
width="500px"
primaryButtonName="Upload"
centered
show={show}
title={<h4>{t('CSV Upload')}</h4>}
>
<AntdForm
form={form}
onFinish={onFinish}
data-test="dashboard-edit-properties-form"
layout="vertical"
initialValues={defaultUploadInfo}
>
<Collapse
expandIconPosition="right"
accordion
defaultActiveKey="general"
css={(theme: SupersetTheme) => antdCollapseStyles(theme)}
>
<Collapse.Panel
header={
<div>
<h4>{t('General information')}</h4>
<p className="helper">
{t('Upload a CSV file to a database.')}
</p>
</div>
}
key="general"
>
<Row>
<Col span={24}>
<StyledFormItem
label={t('CSV File')}
name="upload"
required
rules={[{ validator: validateUpload }]}
>
<Upload
name="modelFile"
id="modelFile"
data-test="model-file-input"
accept={allowedExtensionsToAccept}
fileList={fileList}
onChange={onChangeFile}
onRemove={onRemoveFile}
// upload is handled by hook
customRequest={() => {}}
>
<Button aria-label={t('Select')} icon={<UploadOutlined />}>
{t('Select')}
</Button>
</Upload>
</StyledFormItem>
</Col>
</Row>
<Row>
<Col span={24}>
<ColumnsPreview columns={columns} />
</Col>
</Row>
<Row>
<Col span={24}>
<StyledFormItem
label={t('Database')}
name="database"
required
rules={[{ validator: validateDatabase }]}
>
<AsyncSelect
ariaLabel={t('Select a database')}
options={loadDatabaseOptions}
onChange={onChangeDatabase}
allowClear
placeholder={t('Select a database to upload the file to')}
/>
</StyledFormItem>
</Col>
</Row>
<Row>
<Col span={24}>
<StyledFormItem label={t('Schema')} name="schema">
<AsyncSelect
ariaLabel={t('Select a schema')}
options={loadSchemaOptions}
onChange={onChangeSchema}
allowClear
placeholder={t(
'Select a schema if the database supports this',
)}
/>
</StyledFormItem>
</Col>
</Row>
<Row>
<Col span={24}>
<StyledFormItem
label={t('Table Name')}
name="table_name"
required
rules={[
{ required: true, message: 'Table name is required' },
]}
>
<Input
aria-label={t('Table Name')}
name="table_name"
data-test="properties-modal-name-input"
type="text"
placeholder={t('Name of table to be created with CSV file')}
/>
</StyledFormItem>
</Col>
</Row>
<Row>
<Col span={24}>
<StyledFormItemWithTip
label={t('Delimiter')}
tip={t('Select a delimiter for this data')}
name="delimiter"
>
<Select
ariaLabel={t('Choose a delimiter')}
options={delimiterOptions}
onChange={onChangeDelimiter}
allowNewOptions
/>
</StyledFormItemWithTip>
</Col>
</Row>
</Collapse.Panel>
<Collapse.Panel
header={
<div>
<h4>{t('File Settings')}</h4>
<p className="helper">
{t(
'Adjust how spaces, blank lines, null values are handled and other file wide settings.',
)}
</p>
</div>
}
key="2"
>
<Row>
<Col span={24}>
<StyledFormItemWithTip
label={t('If Table Already Exists')}
tip={t('What should happen if the table already exists')}
name="already_exists"
>
<Select
ariaLabel={t('Choose already exists')}
options={tableAlreadyExistsOptions}
onChange={() => {}}
/>
</StyledFormItemWithTip>
</Col>
</Row>
<Row>
<Col span={24}>
<StyledFormItem
label={t('Columns To Be Parsed as Dates')}
name="column_dates"
>
<Select
ariaLabel={t('Choose columns to be parsed as dates')}
mode="multiple"
options={columnsToOptions()}
allowClear
allowNewOptions
placeholder={t(
'A comma separated list of columns that should be parsed as dates',
)}
/>
</StyledFormItem>
</Col>
</Row>
<Row>
<Col span={24}>
<StyledFormItemWithTip
label={t('Decimal Character')}
tip={t('Character to interpret as decimal point')}
name="decimal_character"
>
<Input type="text" />
</StyledFormItemWithTip>
</Col>
</Row>
<Row>
<Col span={24}>
<StyledFormItemWithTip
label={t('Null Values')}
tip={t(
'Choose values that should be treated as null. Warning: Hive database supports only a single value',
)}
name="null_values"
>
<Select
mode="multiple"
options={nullValuesOptions}
allowClear
allowNewOptions
/>
</StyledFormItemWithTip>
</Col>
</Row>
<Row>
<Col span={24}>
<StyledFormItem name="skip_initial_space">
<SwitchContainer
label={t('Skip spaces after delimiter')}
dataTest="skipInitialSpace"
/>
</StyledFormItem>
</Col>
</Row>
<Row>
<Col span={24}>
<StyledFormItem name="skip_blank_lines">
<SwitchContainer
label={t(
'Skip blank lines rather than interpreting them as Not A Number values',
)}
dataTest="skipBlankLines"
/>
</StyledFormItem>
</Col>
</Row>
<Row>
<Col span={24}>
<StyledFormItem name="day_first">
<SwitchContainer
label={t(
'DD/MM format dates, international and European format',
)}
dataTest="dayFirst"
/>
</StyledFormItem>
</Col>
</Row>
</Collapse.Panel>
<Collapse.Panel
header={
<div>
<h4>{t('Columns')}</h4>
<p className="helper">
{t(
'Adjust column settings such as specifying the columns to read, how duplicates are handled, column data types, and more.',
)}
</p>
</div>
}
key="3"
>
<Row>
<Col span={24}>
<StyledFormItemWithTip
label={t('Index Column')}
tip={t(
'Column to use as the row labels of the dataframe. Leave empty if no index column',
)}
name="index_column"
>
<Select
ariaLabel={t('Choose index column')}
options={columns.map(column => ({
value: column,
label: column,
}))}
allowClear
allowNewOptions
/>
</StyledFormItemWithTip>
</Col>
</Row>
<Row>
<Col span={24}>
<StyledFormItemWithTip
label={t('Column Label(s)')}
tip={t(
'Column label for index column(s). If None is given and Dataframe Index is checked, Index Names are used',
)}
name="column_labels"
>
<Input aria-label={t('Column labels')} type="text" />
</StyledFormItemWithTip>
</Col>
</Row>
<Row>
<Col span={24}>
<StyledFormItem
label={t('Columns To Read')}
name="columns_read"
>
<Select
ariaLabel={t('Choose columns to read')}
mode="multiple"
options={columnsToOptions()}
allowClear
allowNewOptions
placeholder={t(
'List of the column names that should be read',
)}
/>
</StyledFormItem>
</Col>
</Row>
<Row>
<Col span={24}>
<StyledFormItemWithTip
label={t('Column Data Types')}
tip={t(
'A dictionary with column names and their data types if you need to change the defaults. Example: {"user_id":"int"}. Check Python\'s Pandas library for supported data types.',
)}
name="column_data_types"
>
<Input aria-label={t('Column data types')} type="text" />
</StyledFormItemWithTip>
</Col>
</Row>
<Row>
<Col span={24}>
<StyledFormItem name="dataframe_index">
<SwitchContainer
label={t('Write dataframe index as a column')}
dataTest="dataFrameIndex"
/>
</StyledFormItem>
</Col>
</Row>
<Row>
<Col span={24}>
<StyledFormItem name="overwrite_duplicates">
<SwitchContainer
label={t(
'Overwrite Duplicate Columns. If duplicate columns are not overridden, they will be presented as "X.1, X.2 ...X.x"',
)}
dataTest="overwriteDuplicates"
/>
</StyledFormItem>
</Col>
</Row>
</Collapse.Panel>
<Collapse.Panel
header={
<div>
<h4>{t('Rows')}</h4>
<p className="helper">
{t('Set header rows and the number of rows to read or skip.')}
</p>
</div>
}
key="4"
>
<Row>
<Col span={8}>
<StyledFormItemWithTip
label={t('Header Row')}
tip={t(
'Row containing the headers to use as column names (0 is first line of data).',
)}
name="header_row"
rules={[
{ required: true, message: 'Header row is required' },
]}
>
<InputNumber
aria-label={t('Header row')}
type="text"
min={0}
/>
</StyledFormItemWithTip>
</Col>
<Col span={8}>
<StyledFormItemWithTip
label={t('Rows to Read')}
tip={t(
'Number of rows of file to read. Leave empty (default) to read all rows',
)}
name="rows_to_read"
>
<InputNumber aria-label={t('Rows to read')} min={1} />
</StyledFormItemWithTip>
</Col>
<Col span={8}>
<StyledFormItemWithTip
label={t('Skip Rows')}
tip={t('Number of rows to skip at start of file.')}
name="skip_rows"
rules={[{ required: true, message: 'Skip rows is required' }]}
>
<InputNumber aria-label={t('Skip rows')} min={0} />
</StyledFormItemWithTip>
</Col>
</Row>
</Collapse.Panel>
</Collapse>
</AntdForm>
</Modal>
);
};
export default withToasts(CSVUploadModal);

100
superset-frontend/src/features/databases/CSVUploadModal/styles.ts Normal file
View File

@ -0,0 +1,100 @@
/**
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
import { FormItem } from 'src/components/Form';
import { css, styled, SupersetTheme } from '@superset-ui/core';
const MODAL_BODY_HEIGHT = 180.5;
const antIconHeight = 12;
export const StyledFormItem = styled(FormItem)`
${({ theme }) => css`
flex: 1;
margin-top: 0;
margin-bottom: ${theme.gridUnit * 2.5}px;
}
`}
`;
export const StyledSwitchContainer = styled.div`
display: flex;
align-items: center;
margin-top: 0;
`;
export const antdCollapseStyles = (theme: SupersetTheme) => css`
.ant-collapse-header {
padding-top: ${theme.gridUnit * 3.5}px;
padding-bottom: ${theme.gridUnit * 2.5}px;
.anticon.ant-collapse-arrow {
top: calc(50% - ${antIconHeight / 2}px);
}
.helper {
color: ${theme.colors.grayscale.base};
font-size: ${theme.typography.sizes.s}px;
}
}
h4 {
font-size: ${theme.typography.sizes.l}px;
margin-top: 0;
margin-bottom: ${theme.gridUnit}px;
}
p.helper {
margin-bottom: 0;
padding: 0;
}
`;
export const antDModalNoPaddingStyles = css`
.ant-modal-body {
padding-left: 0;
padding-right: 0;
padding-top: 0;
}
`;
export const formStyles = (theme: SupersetTheme) => css`
.switch-label {
color: ${theme.colors.grayscale.base};
margin-left: ${theme.gridUnit * 4}px;
}
`;
export const antDModalStyles = (theme: SupersetTheme) => css`
.ant-modal-header {
padding: ${theme.gridUnit * 4.5}px ${theme.gridUnit * 4}px
${theme.gridUnit * 4}px;
}
.ant-modal-close-x .close {
color: ${theme.colors.grayscale.dark1};
opacity: 1;
}
.ant-modal-body {
height: ${theme.gridUnit * MODAL_BODY_HEIGHT}px;
}
.ant-modal-footer {
height: ${theme.gridUnit * 16.25}px;
}
.info-solid-small {
vertical-align: bottom;
}
`;

2
superset-frontend/src/features/home/RightMenu.test.tsx
View File

@ -175,7 +175,7 @@ const resetUseSelectorMock = () => {
permissions: {},
roles: {
Admin: [
['can_this_form_get', 'CsvToDatabaseView'], // So we can upload CSV
['can_csv_upload', 'Database'], // So we can upload CSV
['can_write', 'Database'], // So we can write DBs
['can_write', 'Dataset'], // So we can write Datasets
['can_write', 'Chart'], // So we can write Datasets

18
superset-frontend/src/features/home/SubMenu.tsx
View File

@ -162,10 +162,14 @@ const StyledHeader = styled.div`
const styledDisabled = (theme: SupersetTheme) => css`
color: ${theme.colors.grayscale.light1};
cursor: not-allowed;
.ant-menu-item-active {
&:hover {
color: ${theme.colors.grayscale.light1};
cursor: default;
}
.ant-menu-item-selected {
background-color: ${theme.colors.grayscale.light1};
}
`;
@ -297,7 +301,11 @@ const SubMenuComponent: React.FunctionComponent<SubMenuProps> = props => {
{link.childs?.map(item => {
if (typeof item === 'object') {
return item.disable ? (
<DropdownMenu.Item key={item.label} css={styledDisabled}>
<DropdownMenu.Item
key={item.label}
css={styledDisabled}
disabled
>
<Tooltip
placement="top"
title={t(
@ -309,7 +317,9 @@ const SubMenuComponent: React.FunctionComponent<SubMenuProps> = props => {
</DropdownMenu.Item>
) : (
<DropdownMenu.Item key={item.label}>
<a href={item.url}>{item.label}</a>
<a href={item.url} onClick={item.onClick}>
{item.label}
</a>
</DropdownMenu.Item>
);
}

17
superset-frontend/src/pages/DatabaseList/index.tsx
View File

@ -49,6 +49,7 @@ import { ExtensionConfigs } from 'src/features/home/types';
import { UserWithPermissionsAndRoles } from 'src/types/bootstrapTypes';
import type { MenuObjectProps } from 'src/types/bootstrapTypes';
import DatabaseModal from 'src/features/databases/DatabaseModal';
import CSVUploadModal from 'src/features/databases/CSVUploadModal';
import { DatabaseObject } from 'src/features/databases/types';
import { ModifiedInfo } from 'src/components/AuditInfo';
import { QueryObjectColumns } from 'src/views/CRUD/types';
@ -135,6 +136,8 @@ function DatabaseList({
const [currentDatabase, setCurrentDatabase] = useState<DatabaseObject | null>(
null,
);
const [csvUploadModalOpen, setCsvUploadModalOpen] = useState<boolean>(false);
const [allowUploads, setAllowUploads] = useState<boolean>(false);
const isAdmin = isUserAdmin(fullUser);
const showUploads = allowUploads || isAdmin;
@ -233,7 +236,10 @@ function DatabaseList({
{
label: t('Upload CSV'),
name: 'Upload CSV file',
url: '/csvtodatabaseview/form',
url: '#',
onClick: () => {
setCsvUploadModalOpen(true);
},
perm: canUploadCSV && showUploads,
disable: isDisabled,
},
@ -557,6 +563,15 @@ function DatabaseList({
refreshData();
}}
/>
<CSVUploadModal
addDangerToast={addDangerToast}
addSuccessToast={addSuccessToast}
onHide={() => {
setCsvUploadModalOpen(false);
}}
show={csvUploadModalOpen}
allowedExtensions={CSV_EXTENSIONS}
/>
{databaseCurrentlyDeleting && (
<DeleteModal
description={

1
superset-frontend/src/types/bootstrapTypes.ts
View File

@ -118,6 +118,7 @@ export interface MenuObjectChildProps {
icon?: string;
index?: number;
url?: string;
onClick?: () => void;
isFrontendRoute?: boolean;
perm?: string | boolean;
view?: string;

2
superset-frontend/src/views/CRUD/utils.tsx
View File

@ -487,7 +487,7 @@ export const uploadUserPerms = (
allowedExt: Array<string>,
) => {
const canUploadCSV =
findPermission('can_this_form_get', 'CsvToDatabaseView', roles) &&
findPermission('can_csv_upload', 'Database', roles) &&
checkUploadExtensions(csvExt, allowedExt);
const canUploadColumnar =
checkUploadExtensions(colExt, allowedExt) &&

198
superset/commands/database/csv_import.py Normal file
View File

@ -0,0 +1,198 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
import logging
from typing import Any, Optional, TypedDict
import pandas as pd
from flask_babel import lazy_gettext as _
from sqlalchemy.exc import SQLAlchemyError
from superset import db
from superset.commands.base import BaseCommand
from superset.commands.database.exceptions import (
DatabaseNotFoundError,
DatabaseSchemaUploadNotAllowed,
DatabaseUploadFailed,
DatabaseUploadSaveMetadataFailed,
)
from superset.connectors.sqla.models import SqlaTable
from superset.daos.database import DatabaseDAO
from superset.models.core import Database
from superset.sql_parse import Table
from superset.utils.core import get_user
from superset.views.database.validators import schema_allows_file_upload
logger = logging.getLogger(__name__)
READ_CSV_CHUNK_SIZE = 1000
class CSVImportOptions(TypedDict, total=False):
schema: str
delimiter: str
already_exists: str
column_data_types: dict[str, str]
column_dates: list[str]
column_labels: str
columns_read: list[str]
dataframe_index: str
day_first: bool
decimal_character: str
header_row: int
index_column: str
null_values: list[str]
overwrite_duplicates: bool
rows_to_read: int
skip_blank_lines: bool
skip_initial_space: bool
skip_rows: bool
class CSVImportCommand(BaseCommand):
def __init__(
self,
model_id: int,
table_name: str,
file: Any,
options: CSVImportOptions,
) -> None:
self._model_id = model_id
self._model: Optional[Database] = None
self._table_name = table_name
self._schema = options.get("schema")
self._file = file
self._options = options
def _read_csv(self) -> pd.DataFrame:
"""
Read CSV file into a DataFrame
:return: pandas DataFrame
:throws DatabaseUploadFailed: if there is an error reading the CSV file
"""
try:
return pd.concat(
pd.read_csv(
chunksize=READ_CSV_CHUNK_SIZE,
encoding="utf-8",
filepath_or_buffer=self._file,
header=self._options.get("header_row", 0),
index_col=self._options.get("index_column"),
dayfirst=self._options.get("day_first", False),
iterator=True,
keep_default_na=not self._options.get("null_values"),
usecols=self._options.get("columns_read")
if self._options.get("columns_read") # None if an empty list
else None,
na_values=self._options.get("null_values")
if self._options.get("null_values") # None if an empty list
else None,
nrows=self._options.get("rows_to_read"),
parse_dates=self._options.get("column_dates"),
sep=self._options.get("delimiter", ","),
skip_blank_lines=self._options.get("skip_blank_lines", False),
skipinitialspace=self._options.get("skip_initial_space", False),
skiprows=self._options.get("skip_rows", 0),
dtype=self._options.get("column_data_types")
if self._options.get("column_data_types")
else None,
)
)
except (
pd.errors.ParserError,
pd.errors.EmptyDataError,
UnicodeDecodeError,
ValueError,
) as ex:
raise DatabaseUploadFailed(
message=_("Parsing error: %(error)s", error=str(ex))
) from ex
except Exception as ex:
raise DatabaseUploadFailed(_("Error reading CSV file")) from ex
def _dataframe_to_database(self, df: pd.DataFrame, database: Database) -> None:
"""
Upload DataFrame to database
:param df:
:throws DatabaseUploadFailed: if there is an error uploading the DataFrame
"""
try:
csv_table = Table(table=self._table_name, schema=self._schema)
database.db_engine_spec.df_to_sql(
database,
csv_table,
df,
to_sql_kwargs={
"chunksize": READ_CSV_CHUNK_SIZE,
"if_exists": self._options.get("already_exists", "fail"),
"index": self._options.get("index_column"),
"index_label": self._options.get("column_labels"),
},
)
except ValueError as ex:
raise DatabaseUploadFailed(
message=_(
"Table already exists. You can change your "
"'if table already exists' strategy to append or "
"replace or provide a different Table Name to use."
)
) from ex
except Exception as ex:
raise DatabaseUploadFailed(exception=ex) from ex
def run(self) -> None:
self.validate()
if not self._model:
return
df = self._read_csv()
self._dataframe_to_database(df, self._model)
sqla_table = (
db.session.query(SqlaTable)
.filter_by(
table_name=self._table_name,
schema=self._schema,
database_id=self._model_id,
)
.one_or_none()
)
if not sqla_table:
sqla_table = SqlaTable(
table_name=self._table_name,
database=self._model,
database_id=self._model_id,
owners=[get_user()],
schema=self._schema,
)
db.session.add(sqla_table)
sqla_table.fetch_metadata()
try:
db.session.commit()
except SQLAlchemyError as ex:
db.session.rollback()
raise DatabaseUploadSaveMetadataFailed() from ex
def validate(self) -> None:
self._model = DatabaseDAO.find_by_id(self._model_id)
if not self._model:
raise DatabaseNotFoundError()
if not schema_allows_file_upload(self._model, self._schema):
raise DatabaseSchemaUploadNotAllowed()

16
superset/commands/database/exceptions.py
View File

@ -89,9 +89,25 @@ class DatabaseExtraValidationError(ValidationError):
class DatabaseNotFoundError(CommandException):
status = 404
message = _("Database not found.")
class DatabaseSchemaUploadNotAllowed(CommandException):
status = 403
message = _("Database schema is not allowed for csv uploads.")
class DatabaseUploadFailed(CommandException):
status = 422
message = _("Database upload file failed")
class DatabaseUploadSaveMetadataFailed(CommandException):
status = 500
message = _("Database upload file failed, while saving metadata")
class DatabaseCreateFailedError(CreateFailedError):
message = _("Database could not be created.")

63
superset/databases/api.py
View File

@ -31,6 +31,7 @@ from sqlalchemy.exc import NoSuchTableError, OperationalError, SQLAlchemyError
from superset import app, event_logger
from superset.commands.database.create import CreateDatabaseCommand
from superset.commands.database.csv_import import CSVImportCommand
from superset.commands.database.delete import DeleteDatabaseCommand
from superset.commands.database.exceptions import (
DatabaseConnectionFailedError,
@ -66,6 +67,7 @@ from superset.daos.database import DatabaseDAO, DatabaseUserOAuth2TokensDAO
from superset.databases.decorators import check_table_access
from superset.databases.filters import DatabaseFilter, DatabaseUploadEnabledFilter
from superset.databases.schemas import (
CSVUploadPostSchema,
database_schemas_query_schema,
database_tables_query_schema,
DatabaseConnectionSchema,
@ -130,6 +132,7 @@ class DatabaseRestApi(BaseSupersetModelRestApi):
"delete_ssh_tunnel",
"schemas_access_for_file_upload",
"get_connection",
"csv_upload",
"oauth2",
}
@ -241,6 +244,7 @@ class DatabaseRestApi(BaseSupersetModelRestApi):
openapi_spec_tag = "Database"
openapi_spec_component_schemas = (
CSVUploadPostSchema,
DatabaseConnectionSchema,
DatabaseFunctionNamesResponse,
DatabaseSchemaAccessForFileUploadResponse,
@ -1336,6 +1340,65 @@ class DatabaseRestApi(BaseSupersetModelRestApi):
command.run()
return self.response(200, message="OK")
@expose("/<int:pk>/csv_upload/", methods=("POST",))
@protect()
@statsd_metrics
@event_logger.log_this_with_context(
action=lambda self, *args, **kwargs: f"{self.__class__.__name__}.import_",
log_to_statsd=False,
)
@requires_form_data
def csv_upload(self, pk: int) -> Response:
"""Upload a CSV file into a database.
---
post:
summary: Upload a CSV file to a database table
parameters:
- in: path
schema:
type: integer
name: pk
requestBody:
required: true
content:
multipart/form-data:
schema:
$ref: '#/components/schemas/CSVUploadPostSchema'
responses:
200:
description: CSV upload response
content:
application/json:
schema:
type: object
properties:
message:
type: string
400:
$ref: '#/components/responses/400'
401:
$ref: '#/components/responses/401'
404:
$ref: '#/components/responses/404'
422:
$ref: '#/components/responses/422'
500:
$ref: '#/components/responses/500'
"""
try:
request_form = request.form.to_dict()
request_form["file"] = request.files.get("file")
parameters = CSVUploadPostSchema().load(request_form)
CSVImportCommand(
pk,
parameters["table_name"],
parameters["file"],
parameters,
).run()
except ValidationError as error:
return self.response_400(message=error.messages)
return self.response(200, message="OK")
@expose("/<int:pk>/function_names/", methods=("GET",))
@protect()
@safe

186
superset/databases/schemas.py
View File

@ -19,13 +19,24 @@
import inspect
import json
import os
import re
from typing import Any
from flask import current_app
from flask_babel import lazy_gettext as _
from marshmallow import EXCLUDE, fields, pre_load, Schema, validates_schema
from marshmallow.validate import Length, ValidationError
from marshmallow import (
EXCLUDE,
fields,
post_load,
pre_load,
Schema,
validates,
validates_schema,
)
from marshmallow.validate import Length, OneOf, Range, ValidationError
from sqlalchemy import MetaData
from werkzeug.datastructures import FileStorage
from superset import db, is_feature_enabled
from superset.commands.database.exceptions import DatabaseInvalidError
@ -980,6 +991,177 @@ class DatabaseConnectionSchema(Schema):
)
class DelimitedListField(fields.List):
"""
Special marshmallow field for handling delimited lists.
formData expects a string, so we need to deserialize it into a list.
"""
def _deserialize(
self, value: str, attr: Any, data: Any, **kwargs: Any
) -> list[Any]:
try:
values = value.split(",") if value else []
return super()._deserialize(values, attr, data, **kwargs)
except AttributeError as exc:
raise ValidationError(
f"{attr} is not a delimited list it has a non string value {value}."
) from exc
class CSVUploadPostSchema(Schema):
"""
Schema for CSV Upload
"""
file = fields.Raw(
required=True,
metadata={
"description": "The CSV file to upload",
"type": "string",
"format": "text/csv",
},
)
delimiter = fields.String(metadata={"description": "The delimiter of the CSV file"})
already_exists = fields.String(
load_default="fail",
validate=OneOf(choices=("fail", "replace", "append")),
metadata={
"description": "What to do if the table already "
"exists accepts: fail, replace, append"
},
)
column_data_types = fields.String(
metadata={
"description": "A dictionary with column names and "
"their data types if you need to change "
"the defaults. Example: {'user_id':'int'}. "
"Check Python Pandas library for supported data types"
}
)
column_dates = DelimitedListField(
fields.String(),
metadata={
"description": "A list of column names that should be "
"parsed as dates. Example: date,timestamp"
},
)
column_labels = fields.String(
metadata={
"description": "Column label for index column(s). "
"If None is given and Dataframe"
"Index is checked, Index Names are used"
}
)
columns_read = DelimitedListField(
fields.String(),
metadata={"description": "A List of the column names that should be read"},
)
dataframe_index = fields.String(
metadata={
"description": "Column to use as the row labels of the dataframe. "
"Leave empty if no index column"
}
)
day_first = fields.Boolean(
metadata={
"description": "DD/MM format dates, international and European format"
}
)
decimal_character = fields.String(
metadata={
"description": "Character to recognize as decimal point. Default is '.'"
}
)
header_row = fields.Integer(
metadata={
"description": "Row containing the headers to use as column names"
"(0 is first line of data). Leave empty if there is no header row."
}
)
index_column = fields.String(
metadata={
"description": "Column to use as the row labels of the dataframe. "
"Leave empty if no index column"
}
)
null_values = DelimitedListField(
fields.String(),
metadata={
"description": "A list of strings that should be treated as null. "
"Examples: '' for empty strings, 'None', 'N/A',"
"Warning: Hive database supports only a single value"
},
)
overwrite_duplicates = fields.Boolean(
metadata={
"description": "If duplicate columns are not overridden,"
"they will be presented as 'X.1, X.2 ...X.x'."
}
)
rows_to_read = fields.Integer(
metadata={
"description": "Number of rows to read from the file. "
"If None, reads all rows."
},
allow_none=True,
validate=Range(min=1),
)
schema = fields.String(
metadata={"description": "The schema to upload the CSV file to."}
)
skip_blank_lines = fields.Boolean(
metadata={"description": "Skip blank lines in the CSV file."}
)
skip_initial_space = fields.Boolean(
metadata={"description": "Skip spaces after delimiter."}
)
skip_rows = fields.Integer(
metadata={"description": "Number of rows to skip at start of file."}
)
table_name = fields.String(
required=True,
validate=[Length(min=1, max=10000)],
allow_none=False,
metadata={"description": "The name of the table to be created/appended"},
)
@post_load
def convert_column_data_types(
self, data: dict[str, Any], **kwargs: Any
) -> dict[str, Any]:
if "column_data_types" in data and data["column_data_types"]:
try:
data["column_data_types"] = json.loads(data["column_data_types"])
except json.JSONDecodeError as ex:
raise ValidationError(
"Invalid JSON format for column_data_types"
) from ex
return data
@validates("file")
def validate_file_size(self, file: FileStorage) -> None:
file.flush()
size = os.fstat(file.fileno()).st_size
if (
current_app.config["CSV_UPLOAD_MAX_SIZE"] is not None
and size > current_app.config["CSV_UPLOAD_MAX_SIZE"]
):
raise ValidationError([_("File size exceeds the maximum allowed size.")])
@validates("file")
def validate_file_extension(self, file: FileStorage) -> None:
allowed_extensions = current_app.config["ALLOWED_EXTENSIONS"].intersection(
current_app.config["CSV_EXTENSIONS"]
)
matches = re.match(r".+\.([^.]+)$", file.filename)
if not matches:
raise ValidationError([_("File extension is not allowed.")])
extension = matches.group(1)
if extension not in allowed_extensions:
raise ValidationError([_("File extension is not allowed.")])
class OAuth2ProviderResponseSchema(Schema):
"""
Schema for the payload sent on OAuth2 redirect.

25
superset/forms.py
View File

@ -16,12 +16,10 @@
# under the License.
"""Contains the logic to create cohesive forms on the explore view"""
import json
import os
from typing import Any, Optional
from flask_appbuilder.fieldwidgets import BS3TextFieldWidget
from flask_babel import gettext as _
from wtforms import Field, ValidationError
from wtforms import Field
class JsonListField(Field):
@ -55,27 +53,6 @@ class CommaSeparatedListField(Field):
self.data = []
class FileSizeLimit: # pylint: disable=too-few-public-methods
"""Imposes an optional maximum filesize limit for uploaded files"""
def __init__(self, max_size: Optional[int]):
self.max_size = max_size
def __call__(self, form: dict[str, Any], field: Any) -> None:
if self.max_size is None:
return
field.data.flush()
size = os.fstat(field.data.fileno()).st_size
if size > self.max_size:
raise ValidationError(
_(
"File size must be less than or equal to %(max_size)s bytes",
max_size=self.max_size,
)
)
def filter_not_empty_values(values: Optional[list[Any]]) -> Optional[list[Any]]:
"""Returns a list of non empty values or None"""
if not values:

2
superset/initialization/__init__.py
View File

@ -172,7 +172,6 @@ class SupersetAppInitializer: # pylint: disable=too-many-public-methods
)
from superset.views.database.views import (
ColumnarToDatabaseView,
CsvToDatabaseView,
DatabaseView,
ExcelToDatabaseView,
)
@ -295,7 +294,6 @@ class SupersetAppInitializer: # pylint: disable=too-many-public-methods
#
appbuilder.add_view_no_menu(Api)
appbuilder.add_view_no_menu(CssTemplateAsyncModelView)
appbuilder.add_view_no_menu(CsvToDatabaseView)
appbuilder.add_view_no_menu(ExcelToDatabaseView)
appbuilder.add_view_no_menu(ColumnarToDatabaseView)
appbuilder.add_view_no_menu(Dashboard)

85
superset/migrations/versions/2024-04-08_15-43_5ad7321c2169_mig_new_csv_upload_perm.py Normal file
View File

@ -0,0 +1,85 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""mig new csv upload perm
Revision ID: 5ad7321c2169
Revises: 678eefb4ab44
Create Date: 2024-04-08 15:43:29.682687
"""
# revision identifiers, used by Alembic.
revision = "5ad7321c2169"
down_revision = "678eefb4ab44"
from alembic import op
from sqlalchemy.exc import SQLAlchemyError
from sqlalchemy.orm import Session
from superset.migrations.shared.security_converge import (
add_pvms,
get_reversed_new_pvms,
get_reversed_pvm_map,
migrate_roles,
Pvm,
)
NEW_PVMS = {"Database": ("can_csv_upload",)}
PVM_MAP = {
Pvm("CsvToDatabaseView", "can_this_form_post"): (
Pvm("Database", "can_csv_upload"),
),
Pvm("CsvToDatabaseView", "can_this_form_get"): (Pvm("Database", "can_csv_upload"),),
}
def do_upgrade(session: Session) -> None:
add_pvms(session, NEW_PVMS)
migrate_roles(session, PVM_MAP)
def do_downgrade(session: Session) -> None:
add_pvms(session, get_reversed_new_pvms(PVM_MAP))
migrate_roles(session, get_reversed_pvm_map(PVM_MAP))
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
do_upgrade(session)
try:
session.commit()
except SQLAlchemyError as ex:
session.rollback()
raise Exception(f"An error occurred while upgrading permissions: {ex}")
def downgrade():
bind = op.get_bind()
session = Session(bind=bind)
do_downgrade(session)
try:
session.commit()
except SQLAlchemyError as ex:
print(f"An error occurred while downgrading permissions: {ex}")
session.rollback()
pass

11
superset/security/manager.py
View File

@ -242,7 +242,6 @@ class SupersetSecurityManager( # pylint: disable=too-many-public-methods
"CSS Templates",
"ColumnarToDatabaseView",
"CssTemplate",
"CsvToDatabaseView",
"ExcelToDatabaseView",
"Import dashboards",
"ImportExportRestApi",
@ -250,7 +249,10 @@ class SupersetSecurityManager( # pylint: disable=too-many-public-methods
"Queries",
"ReportSchedule",
"TableSchemaView",
"Upload a CSV",
}
ALPHA_ONLY_PMVS = {
("can_csv_upload", "Database"),
}
ADMIN_ONLY_PERMISSIONS = {
@ -997,7 +999,8 @@ class SupersetSecurityManager( # pylint: disable=too-many-public-methods
:param pvm: The FAB permission/view
:returns: Whether the FAB object is accessible to only Admin users
"""
if (pvm.permission.name, pvm.view_menu.name) in self.ALPHA_ONLY_PMVS:
return False
if (
pvm.view_menu.name in self.READ_ONLY_MODEL_VIEWS
and pvm.permission.name not in self.READ_ONLY_PERMISSION
@ -1022,6 +1025,8 @@ class SupersetSecurityManager( # pylint: disable=too-many-public-methods
and pvm.permission.name not in self.READ_ONLY_PERMISSION
):
return True
if (pvm.permission.name, pvm.view_menu.name) in self.ALPHA_ONLY_PMVS:
return True
return (
pvm.view_menu.name in self.ALPHA_ONLY_VIEW_MENUS
or pvm.permission.name in self.ALPHA_ONLY_PERMISSIONS

173
superset/views/database/forms.py
View File

@ -33,7 +33,6 @@ from wtforms.validators import DataRequired, Length, NumberRange, Optional, Rege
from superset import app, db, security_manager
from superset.forms import (
CommaSeparatedListField,
FileSizeLimit,
filter_not_empty_values,
JsonListField,
)
@ -104,178 +103,6 @@ class UploadToDatabaseForm(DynamicForm):
return False
class CsvToDatabaseForm(UploadToDatabaseForm):
csv_file = FileField(
_("CSV Upload"),
description=_("Select a file to be uploaded to the database"),
validators=[
FileRequired(),
FileSizeLimit(config["CSV_UPLOAD_MAX_SIZE"]),
FileAllowed(
config["ALLOWED_EXTENSIONS"].intersection(config["CSV_EXTENSIONS"]),
_(
"Only the following file extensions are allowed: "
"%(allowed_extensions)s",
allowed_extensions=", ".join(
config["ALLOWED_EXTENSIONS"].intersection(
config["CSV_EXTENSIONS"]
)
),
),
),
],
)
table_name = StringField(
_("Table Name"),
description=_("Name of table to be created with CSV file"),
validators=[
DataRequired(),
Regexp(r"^[^\.]+$", message=_("Table name cannot contain a schema")),
],
widget=BS3TextFieldWidget(),
)
database = QuerySelectField(
_("Database"),
description=_("Select a database to upload the file to"),
query_func=UploadToDatabaseForm.file_allowed_dbs,
get_pk_func=lambda a: a.id,
get_label=lambda a: a.database_name,
)
dtype = StringField(
_("Column Data Types"),
description=_(
"A dictionary with column names and their data types"
" if you need to change the defaults."
' Example: {"user_id":"int"}. '
"Check Python's Pandas library for supported data types."
),
validators=[Optional()],
widget=BS3TextFieldWidget(),
)
schema = StringField(
_("Schema"),
description=_("Select a schema if the database supports this"),
validators=[Optional()],
widget=BS3TextFieldWidget(),
)
delimiter = SelectField(
_("Delimiter"),
description=_("Enter a delimiter for this data"),
choices=[
(",", _(",")),
(".", _(".")),
("other", _("Other")),
],
validators=[DataRequired()],
default=[","],
)
otherInput = StringField(
_("Other"),
)
if_exists = SelectField(
_("If Table Already Exists"),
description=_("What should happen if the table already exists"),
choices=[
("fail", _("Fail")),
("replace", _("Replace")),
("append", _("Append")),
],
validators=[DataRequired()],
)
skip_initial_space = BooleanField(
_("Skip Initial Space"), description=_("Skip spaces after delimiter")
)
skip_blank_lines = BooleanField(
_("Skip Blank Lines"),
description=_(
"Skip blank lines rather than interpreting them as Not A Number values"
),
)
parse_dates = CommaSeparatedListField(
_("Columns To Be Parsed as Dates"),
description=_(
"A comma separated list of columns that should be parsed as dates"
),
filters=[filter_not_empty_values],
)
day_first = BooleanField(
_("Day First"),
description=_("DD/MM format dates, international and European format"),
)
decimal = StringField(
_("Decimal Character"),
default=".",
description=_("Character to interpret as decimal point"),
validators=[Optional(), Length(min=1, max=1)],
widget=BS3TextFieldWidget(),
)
null_values = JsonListField(
_("Null Values"),
default=config["CSV_DEFAULT_NA_NAMES"],
description=_(
"Json list of the values that should be treated as null. "
'Examples: [""] for empty strings, ["None", "N/A"], ["nan", "null"]. '
"Warning: Hive database supports only a single value"
),
)
index_col = IntegerField(
_("Index Column"),
description=_(
"Column to use as the row labels of the "
"dataframe. Leave empty if no index column"
),
validators=[Optional(), NumberRange(min=0)],
widget=BS3TextFieldWidget(),
)
dataframe_index = BooleanField(
_("Dataframe Index"), description=_("Write dataframe index as a column")
)
index_label = StringField(
_("Column Label(s)"),
description=_(
"Column label for index column(s). If None is given "
"and Dataframe Index is checked, Index Names are used"
),
validators=[Optional()],
widget=BS3TextFieldWidget(),
)
use_cols = JsonListField(
_("Columns To Read"),
default=None,
description=_("Json list of the column names that should be read"),
validators=[Optional()],
)
overwrite_duplicate = BooleanField(
_("Overwrite Duplicate Columns"),
description=_(
"If duplicate columns are not overridden, "
'they will be presented as "X.1, X.2 ...X.x"'
),
)
header = IntegerField(
_("Header Row"),
description=_(
"Row containing the headers to use as "
"column names (0 is first line of data). "
"Leave empty if there is no header row"
),
validators=[Optional(), NumberRange(min=0)],
widget=BS3TextFieldWidget(),
)
nrows = IntegerField(
_("Rows to Read"),
description=_("Number of rows of file to read"),
validators=[Optional(), NumberRange(min=0)],
widget=BS3TextFieldWidget(),
)
skiprows = IntegerField(
_("Skip Rows"),
description=_("Number of rows to skip at start of file"),
validators=[Optional(), NumberRange(min=0)],
widget=BS3TextFieldWidget(),
)
class ExcelToDatabaseForm(UploadToDatabaseForm):
name = StringField(
_("Table Name"),

147
superset/views/database/views.py
View File

@ -15,7 +15,6 @@
# specific language governing permissions and limitations
# under the License.
import io
import json
import os
import tempfile
import zipfile
@ -42,7 +41,7 @@ from superset.superset_typing import FlaskResponse
from superset.utils import core as utils
from superset.views.base import DeleteMixin, SupersetModelView, YamlExportMixin
from .forms import ColumnarToDatabaseForm, CsvToDatabaseForm, ExcelToDatabaseForm
from .forms import ColumnarToDatabaseForm, ExcelToDatabaseForm
from .mixins import DatabaseMixin
from .validators import schema_allows_file_upload, sqlalchemy_uri_validator
@ -155,150 +154,6 @@ class CustomFormView(SimpleFormView):
)
class CsvToDatabaseView(CustomFormView):
form = CsvToDatabaseForm
form_template = "superset/form_view/csv_to_database_view/edit.html"
form_title = _("CSV to Database configuration")
add_columns = ["database", "schema", "table_name"]
def form_get(self, form: CsvToDatabaseForm) -> None:
form.delimiter.data = ","
form.header.data = 0
form.overwrite_duplicate.data = True
form.skip_initial_space.data = False
form.skip_blank_lines.data = True
form.day_first.data = False
form.decimal.data = "."
form.if_exists.data = "fail"
def form_post(self, form: CsvToDatabaseForm) -> Response:
database = form.database.data
csv_table = Table(table=form.table_name.data, schema=form.schema.data)
delimiter_input = form.delimiter.data
if not schema_allows_file_upload(database, csv_table.schema):
message = _(
'Database "%(database_name)s" schema "%(schema_name)s" '
"is not allowed for csv uploads. Please contact your Superset Admin.",
database_name=database.database_name,
schema_name=csv_table.schema,
)
flash(message, "danger")
return redirect("/csvtodatabaseview/form")
if form.delimiter.data == "other":
delimiter_input = form.otherInput.data
try:
kwargs = {"dtype": json.loads(form.dtype.data)} if form.dtype.data else {}
df = pd.concat(
pd.read_csv(
chunksize=1000,
encoding="utf-8",
filepath_or_buffer=form.csv_file.data,
header=form.header.data if form.header.data else 0,
index_col=form.index_col.data,
dayfirst=form.day_first.data,
iterator=True,
keep_default_na=not form.null_values.data,
usecols=form.use_cols.data if form.use_cols.data else None,
na_values=form.null_values.data if form.null_values.data else None,
nrows=form.nrows.data,
parse_dates=form.parse_dates.data,
sep=delimiter_input,
skip_blank_lines=form.skip_blank_lines.data,
skipinitialspace=form.skip_initial_space.data,
skiprows=form.skiprows.data,
**kwargs,
)
)
database = (
db.session.query(models.Database)
.filter_by(id=form.data.get("database").data.get("id"))
.one()
)
database.db_engine_spec.df_to_sql(
database,
csv_table,
df,
to_sql_kwargs={
"chunksize": 1000,
"if_exists": form.if_exists.data,
"index": form.dataframe_index.data,
"index_label": form.index_label.data,
},
)
# Connect table to the database that should be used for exploration.
# E.g. if hive was used to upload a csv, presto will be a better option
# to explore the table.
explore_database = database
explore_database_id = database.explore_database_id
if explore_database_id:
explore_database = (
db.session.query(models.Database)
.filter_by(id=explore_database_id)
.one_or_none()
or database
)
sqla_table = (
db.session.query(SqlaTable)
.filter_by(
table_name=csv_table.table,
schema=csv_table.schema,
database_id=explore_database.id,
)
.one_or_none()
)
if sqla_table:
sqla_table.fetch_metadata()
if not sqla_table:
sqla_table = SqlaTable(table_name=csv_table.table)
sqla_table.database = explore_database
sqla_table.database_id = database.id
sqla_table.owners = [g.user]
sqla_table.schema = csv_table.schema
sqla_table.fetch_metadata()
db.session.add(sqla_table)
db.session.commit()
except Exception as ex: # pylint: disable=broad-except
db.session.rollback()
message = _(
'Unable to upload CSV file "%(filename)s" to table '
'"%(table_name)s" in database "%(db_name)s". '
"Error message: %(error_msg)s",
filename=form.csv_file.data.filename,
table_name=form.table_name.data,
db_name=database.database_name,
error_msg=str(ex),
)
flash(message, "danger")
stats_logger.incr("failed_csv_upload")
return redirect("/csvtodatabaseview/form")
# Go back to welcome page / splash screen
message = _(
'CSV file "%(csv_filename)s" uploaded to table "%(table_name)s" in '
'database "%(db_name)s"',
csv_filename=form.csv_file.data.filename,
table_name=str(csv_table),
db_name=sqla_table.database.database_name,
)
flash(message, "info")
event_logger.log_with_context(
action="successful_csv_upload",
database=form.database.data.name,
schema=form.schema.data,
table=form.table_name.data,
)
return redirect("/tablemodelview/list/")
class ExcelToDatabaseView(SimpleFormView):
form = ExcelToDatabaseForm
form_template = "superset/form_view/excel_to_database_view/edit.html"

3
tests/integration_tests/core_tests.py
View File

@ -491,9 +491,6 @@ class TestCore(SupersetTestCase):
add_datasource_page = self.get_resp("/databaseview/list/")
self.assertIn("Upload a CSV", add_datasource_page)
form_get = self.get_resp("/csvtodatabaseview/form")
self.assertIn("CSV to Database configuration", form_get)
def test_dataframe_timezone(self):
tz = pytz.FixedOffset(60)
data = [

226
tests/integration_tests/csv_upload_tests.py
View File

@ -20,7 +20,7 @@ import json
import logging
import os
import shutil
from typing import Optional, Union
from typing import Optional
from unittest import mock
@ -129,31 +129,6 @@ def get_upload_db():
return db.session.query(Database).filter_by(database_name=CSV_UPLOAD_DATABASE).one()
def upload_csv(
filename: str,
table_name: str,
extra: Optional[dict[str, str]] = None,
dtype: Union[str, None] = None,
):
csv_upload_db_id = get_upload_db().id
form_data = {
"csv_file": open(filename, "rb"),
"delimiter": ",",
"table_name": table_name,
"database": csv_upload_db_id,
"if_exists": "fail",
"index_label": "test_label",
"overwrite_duplicate": False,
}
if schema := utils.get_example_default_schema():
form_data["schema"] = schema
if extra:
form_data.update(extra)
if dtype:
form_data["dtype"] = dtype
return get_resp(test_client, "/csvtodatabaseview/form", data=form_data)
def upload_excel(
filename: str, table_name: str, extra: Optional[dict[str, str]] = None
):
@ -224,205 +199,6 @@ def escaped_parquet(text):
return escaped_double_quotes(f"[&#39;{text}&#39;]")
@pytest.mark.usefixtures("setup_csv_upload_with_context")
@pytest.mark.usefixtures("create_csv_files")
@mock.patch(
"superset.models.core.config",
{**app.config, "ALLOWED_USER_CSV_SCHEMA_FUNC": lambda d, u: ["admin_database"]},
)
@mock.patch("superset.db_engine_specs.hive.upload_to_s3", mock_upload_to_s3)
@mock.patch("superset.views.database.views.event_logger.log_with_context")
def test_import_csv_enforced_schema(mock_event_logger):
if utils.backend() == "sqlite":
pytest.skip("Sqlite doesn't support schema / database creation")
if utils.backend() == "mysql":
pytest.skip("This test is flaky on MySQL")
full_table_name = f"admin_database.{CSV_UPLOAD_TABLE_W_SCHEMA}"
# Invalid table name
resp = upload_csv(CSV_FILENAME1, full_table_name)
assert "Table name cannot contain a schema" in resp
# no schema specified, fail upload
resp = upload_csv(CSV_FILENAME1, CSV_UPLOAD_TABLE_W_SCHEMA, extra={"schema": None})
assert (
f"Database {escaped_double_quotes(CSV_UPLOAD_DATABASE)} schema"
f" {escaped_double_quotes('None')} is not allowed for csv uploads" in resp
)
success_msg = f"CSV file {escaped_double_quotes(CSV_FILENAME1)} uploaded to table {escaped_double_quotes(full_table_name)}"
resp = upload_csv(
CSV_FILENAME1,
CSV_UPLOAD_TABLE_W_SCHEMA,
extra={"schema": "admin_database", "if_exists": "replace"},
)
assert success_msg in resp
mock_event_logger.assert_called_with(
action="successful_csv_upload",
database=get_upload_db().name,
schema="admin_database",
table=CSV_UPLOAD_TABLE_W_SCHEMA,
)
with get_upload_db().get_sqla_engine() as engine:
data = engine.execute(
f"SELECT * from {ADMIN_SCHEMA_NAME}.{CSV_UPLOAD_TABLE_W_SCHEMA} ORDER BY b"
).fetchall()
assert data == [("john", 1), ("paul", 2)]
# user specified schema doesn't match, fail
resp = upload_csv(
CSV_FILENAME1, CSV_UPLOAD_TABLE_W_SCHEMA, extra={"schema": "gold"}
)
assert (
f'Database {escaped_double_quotes(CSV_UPLOAD_DATABASE)} schema {escaped_double_quotes("gold")} is not allowed for csv uploads'
in resp
)
# user specified schema matches the expected schema, append
if utils.backend() == "hive":
pytest.skip("Hive database doesn't support append csv uploads.")
resp = upload_csv(
CSV_FILENAME1,
CSV_UPLOAD_TABLE_W_SCHEMA,
extra={"schema": "admin_database", "if_exists": "append"},
)
assert success_msg in resp
# Clean up
with get_upload_db().get_sqla_engine() as engine:
engine.execute(f"DROP TABLE {full_table_name}")
@mock.patch("superset.db_engine_specs.hive.upload_to_s3", mock_upload_to_s3)
def test_import_csv_explore_database(setup_csv_upload_with_context, create_csv_files):
schema = utils.get_example_default_schema()
full_table_name = (
f"{schema}.{CSV_UPLOAD_TABLE_W_EXPLORE}"
if schema
else CSV_UPLOAD_TABLE_W_EXPLORE
)
if utils.backend() == "sqlite":
pytest.skip("Sqlite doesn't support schema / database creation")
resp = upload_csv(CSV_FILENAME1, CSV_UPLOAD_TABLE_W_EXPLORE)
assert (
f"CSV file {escaped_double_quotes(CSV_FILENAME1)} uploaded to table {escaped_double_quotes(full_table_name)}"
in resp
)
table = SupersetTestCase.get_table(name=CSV_UPLOAD_TABLE_W_EXPLORE)
assert table.database_id == superset.utils.database.get_example_database().id
@pytest.mark.usefixtures("setup_csv_upload_with_context")
@pytest.mark.usefixtures("create_csv_files")
@mock.patch("superset.db_engine_specs.hive.upload_to_s3", mock_upload_to_s3)
@mock.patch("superset.views.database.views.event_logger.log_with_context")
def test_import_csv(mock_event_logger):
schema = utils.get_example_default_schema()
full_table_name = f"{schema}.{CSV_UPLOAD_TABLE}" if schema else CSV_UPLOAD_TABLE
success_msg_f1 = f"CSV file {escaped_double_quotes(CSV_FILENAME1)} uploaded to table {escaped_double_quotes(full_table_name)}"
test_db = get_upload_db()
# initial upload with fail mode
resp = upload_csv(CSV_FILENAME1, CSV_UPLOAD_TABLE)
assert success_msg_f1 in resp
# upload again with fail mode; should fail
fail_msg = f"Unable to upload CSV file {escaped_double_quotes(CSV_FILENAME1)} to table {escaped_double_quotes(CSV_UPLOAD_TABLE)}"
resp = upload_csv(CSV_FILENAME1, CSV_UPLOAD_TABLE)
assert fail_msg in resp
if utils.backend() != "hive":
# upload again with append mode
resp = upload_csv(
CSV_FILENAME1, CSV_UPLOAD_TABLE, extra={"if_exists": "append"}
)
assert success_msg_f1 in resp
mock_event_logger.assert_called_with(
action="successful_csv_upload",
database=test_db.name,
schema=schema,
table=CSV_UPLOAD_TABLE,
)
# upload again with replace mode
resp = upload_csv(CSV_FILENAME1, CSV_UPLOAD_TABLE, extra={"if_exists": "replace"})
assert success_msg_f1 in resp
# try to append to table from file with different schema
resp = upload_csv(CSV_FILENAME2, CSV_UPLOAD_TABLE, extra={"if_exists": "append"})
fail_msg_f2 = f"Unable to upload CSV file {escaped_double_quotes(CSV_FILENAME2)} to table {escaped_double_quotes(CSV_UPLOAD_TABLE)}"
assert fail_msg_f2 in resp
# replace table from file with different schema
resp = upload_csv(CSV_FILENAME2, CSV_UPLOAD_TABLE, extra={"if_exists": "replace"})
success_msg_f2 = f"CSV file {escaped_double_quotes(CSV_FILENAME2)} uploaded to table {escaped_double_quotes(full_table_name)}"
assert success_msg_f2 in resp
table = SupersetTestCase.get_table(name=CSV_UPLOAD_TABLE)
# make sure the new column name is reflected in the table metadata
assert "d" in table.column_names
# ensure user is assigned as an owner
assert security_manager.find_user("admin") in table.owners
# null values are set
upload_csv(
CSV_FILENAME2,
CSV_UPLOAD_TABLE,
extra={"null_values": '["", "john"]', "if_exists": "replace"},
)
# make sure that john and empty string are replaced with None
with test_db.get_sqla_engine() as engine:
data = engine.execute(f"SELECT * from {CSV_UPLOAD_TABLE} ORDER BY c").fetchall()
assert data == [(None, 1, "x"), ("paul", 2, None)]
# default null values
upload_csv(CSV_FILENAME2, CSV_UPLOAD_TABLE, extra={"if_exists": "replace"})
# make sure that john and empty string are replaced with None
data = engine.execute(f"SELECT * from {CSV_UPLOAD_TABLE} ORDER BY c").fetchall()
assert data == [("john", 1, "x"), ("paul", 2, None)]
# cleanup
with get_upload_db().get_sqla_engine() as engine:
engine.execute(f"DROP TABLE {full_table_name}")
# with dtype
upload_csv(
CSV_FILENAME1,
CSV_UPLOAD_TABLE,
dtype='{"a": "string", "b": "float64"}',
)
# you can change the type to something compatible, like an object to string
# or an int to a float
# file upload should work as normal
with test_db.get_sqla_engine() as engine:
data = engine.execute(f"SELECT * from {CSV_UPLOAD_TABLE} ORDER BY b").fetchall()
assert data == [("john", 1), ("paul", 2)]
# cleanup
with get_upload_db().get_sqla_engine() as engine:
engine.execute(f"DROP TABLE {full_table_name}")
# with dtype - wrong type
resp = upload_csv(
CSV_FILENAME1,
CSV_UPLOAD_TABLE,
dtype='{"a": "int"}',
)
# you cannot pass an incompatible dtype
fail_msg = f"Unable to upload CSV file {escaped_double_quotes(CSV_FILENAME1)} to table {escaped_double_quotes(CSV_UPLOAD_TABLE)}"
assert fail_msg in resp
@pytest.mark.usefixtures("setup_csv_upload_with_context")
@pytest.mark.usefixtures("create_excel_files")
@mock.patch("superset.db_engine_specs.hive.upload_to_s3", mock_upload_to_s3)

7
tests/integration_tests/databases/api_tests.py
View File

@ -1445,7 +1445,12 @@ class TestDatabaseApi(SupersetTestCase):
rv = self.get_assert_metric(uri, "info")
data = json.loads(rv.data.decode("utf-8"))
assert rv.status_code == 200
assert set(data["permissions"]) == {"can_read", "can_write", "can_export"}
assert set(data["permissions"]) == {
"can_read",
"can_csv_upload",
"can_write",
"can_export",
}
def test_get_invalid_database_table_metadata(self):
"""

16
tests/integration_tests/databases/commands/__init__.py Normal file
View File

@ -0,0 +1,16 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

283
tests/integration_tests/databases/commands/csv_upload_test.py Normal file
View File

@ -0,0 +1,283 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
import json
from datetime import datetime
import pytest
from superset import db, security_manager
from superset.commands.database.csv_import import CSVImportCommand
from superset.commands.database.exceptions import (
DatabaseNotFoundError,
DatabaseSchemaUploadNotAllowed,
DatabaseUploadFailed,
)
from superset.models.core import Database
from superset.utils.core import override_user
from superset.utils.database import get_or_create_db
from tests.integration_tests.conftest import only_postgresql
from tests.integration_tests.test_app import app
from tests.unit_tests.fixtures.common import create_csv_file
CSV_UPLOAD_DATABASE = "csv_explore_db"
CSV_UPLOAD_TABLE = "csv_upload"
CSV_UPLOAD_TABLE_W_SCHEMA = "csv_upload_w_schema"
CSV_FILE_1 = [
["Name", "Age", "City", "Birth"],
["name1", "30", "city1", "1-1-1980"],
["name2", "29", "city2", "1-1-1981"],
["name3", "28", "city3", "1-1-1982"],
]
CSV_FILE_2 = [
["name1", "30", "city1", "1-1-1980"],
["Name", "Age", "City", "Birth"],
["name2", "29", "city2", "1-1-1981"],
["name3", "28", "city3", "1-1-1982"],
]
CSV_FILE_3 = [
["Name", "Age", "City", "Birth"],
["name1", "N/A", "city1", "1-1-1980"],
["name2", "29", "None", "1-1-1981"],
["name3", "28", "city3", "1-1-1982"],
]
CSV_FILE_BROKEN = [
["Name", "Age", "City", "Birth"],
["name1", "30", "city1", "1-1-1980"],
["name2", "29"],
["name3", "28", "city3", "1-1-1982"],
]
def _setup_csv_upload(allowed_schemas: list[str] | None = None):
upload_db = get_or_create_db(
CSV_UPLOAD_DATABASE, app.config["SQLALCHEMY_EXAMPLES_URI"]
)
upload_db.allow_file_upload = True
extra = upload_db.get_extra()
allowed_schemas = allowed_schemas or []
extra["schemas_allowed_for_file_upload"] = allowed_schemas
upload_db.extra = json.dumps(extra)
db.session.commit()
yield
upload_db = get_upload_db()
with upload_db.get_sqla_engine_with_context() as engine:
engine.execute(f"DROP TABLE IF EXISTS {CSV_UPLOAD_TABLE}")
engine.execute(f"DROP TABLE IF EXISTS {CSV_UPLOAD_TABLE_W_SCHEMA}")
db.session.delete(upload_db)
db.session.commit()
def get_upload_db():
return db.session.query(Database).filter_by(database_name=CSV_UPLOAD_DATABASE).one()
@pytest.fixture(scope="function")
def setup_csv_upload_with_context():
with app.app_context():
yield from _setup_csv_upload()
@pytest.fixture(scope="function")
def setup_csv_upload_with_context_schema():
with app.app_context():
yield from _setup_csv_upload(["public"])
@only_postgresql
@pytest.mark.parametrize(
"csv_data,options, table_data",
[
(
CSV_FILE_1,
{},
[
("name1", 30, "city1", "1-1-1980"),
("name2", 29, "city2", "1-1-1981"),
("name3", 28, "city3", "1-1-1982"),
],
),
(
CSV_FILE_1,
{"columns_read": ["Name", "Age"]},
[("name1", 30), ("name2", 29), ("name3", 28)],
),
(
CSV_FILE_1,
{"columns_read": []},
[
("name1", 30, "city1", "1-1-1980"),
("name2", 29, "city2", "1-1-1981"),
("name3", 28, "city3", "1-1-1982"),
],
),
(
CSV_FILE_1,
{"rows_to_read": 1},
[
("name1", 30, "city1", "1-1-1980"),
],
),
(
CSV_FILE_1,
{"rows_to_read": 1, "columns_read": ["Name", "Age"]},
[
("name1", 30),
],
),
(
CSV_FILE_1,
{"skip_rows": 1},
[("name2", 29, "city2", "1-1-1981"), ("name3", 28, "city3", "1-1-1982")],
),
(
CSV_FILE_1,
{"rows_to_read": 2},
[
("name1", 30, "city1", "1-1-1980"),
("name2", 29, "city2", "1-1-1981"),
],
),
(
CSV_FILE_1,
{"column_dates": ["Birth"]},
[
("name1", 30, "city1", datetime(1980, 1, 1, 0, 0)),
("name2", 29, "city2", datetime(1981, 1, 1, 0, 0)),
("name3", 28, "city3", datetime(1982, 1, 1, 0, 0)),
],
),
(
CSV_FILE_2,
{"header_row": 1},
[("name2", 29, "city2", "1-1-1981"), ("name3", 28, "city3", "1-1-1982")],
),
(
CSV_FILE_3,
{"null_values": ["N/A", "None"]},
[
("name1", None, "city1", "1-1-1980"),
("name2", 29, None, "1-1-1981"),
("name3", 28, "city3", "1-1-1982"),
],
),
(
CSV_FILE_3,
{
"null_values": ["N/A", "None"],
"column_dates": ["Birth"],
"columns_read": ["Name", "Age", "Birth"],
},
[
("name1", None, datetime(1980, 1, 1, 0, 0)),
("name2", 29, datetime(1981, 1, 1, 0, 0)),
("name3", 28, datetime(1982, 1, 1, 0, 0)),
],
),
(
CSV_FILE_BROKEN,
{},
[
("name1", 30, "city1", "1-1-1980"),
("name2", 29, None, None),
("name3", 28, "city3", "1-1-1982"),
],
),
],
)
@pytest.mark.usefixtures("setup_csv_upload_with_context")
def test_csv_upload_options(csv_data, options, table_data):
admin_user = security_manager.find_user(username="admin")
upload_database = get_upload_db()
with override_user(admin_user):
CSVImportCommand(
upload_database.id,
CSV_UPLOAD_TABLE,
create_csv_file(csv_data),
options=options,
).run()
with upload_database.get_sqla_engine_with_context() as engine:
data = engine.execute(f"SELECT * from {CSV_UPLOAD_TABLE}").fetchall()
assert data == table_data
@only_postgresql
@pytest.mark.usefixtures("setup_csv_upload_with_context")
def test_csv_upload_database_not_found():
admin_user = security_manager.find_user(username="admin")
with override_user(admin_user):
with pytest.raises(DatabaseNotFoundError):
CSVImportCommand(
1000,
CSV_UPLOAD_TABLE,
create_csv_file(CSV_FILE_1),
options={},
).run()
@only_postgresql
@pytest.mark.usefixtures("setup_csv_upload_with_context_schema")
def test_csv_upload_schema_not_allowed():
admin_user = security_manager.find_user(username="admin")
upload_db_id = get_upload_db().id
with override_user(admin_user):
with pytest.raises(DatabaseSchemaUploadNotAllowed):
CSVImportCommand(
upload_db_id,
CSV_UPLOAD_TABLE,
create_csv_file(CSV_FILE_1),
options={},
).run()
with pytest.raises(DatabaseSchemaUploadNotAllowed):
CSVImportCommand(
upload_db_id,
CSV_UPLOAD_TABLE,
create_csv_file(CSV_FILE_1),
options={"schema": "schema1"},
).run()
CSVImportCommand(
upload_db_id,
CSV_UPLOAD_TABLE,
create_csv_file(CSV_FILE_1),
options={"schema": "public"},
).run()
@only_postgresql
@pytest.mark.usefixtures("setup_csv_upload_with_context")
def test_csv_upload_broken_file():
admin_user = security_manager.find_user(username="admin")
with override_user(admin_user):
with pytest.raises(DatabaseUploadFailed):
CSVImportCommand(
get_upload_db().id,
CSV_UPLOAD_TABLE,
create_csv_file([""]),
options={"column_dates": ["Birth"]},
).run()

4
tests/integration_tests/security_tests.py
View File

@ -1323,6 +1323,7 @@ class TestRolePermission(SupersetTestCase):
self.assert_cannot_menu("Upload a CSV", perm_set)
self.assert_cannot_menu("ReportSchedule", perm_set)
self.assert_cannot_menu("Alerts & Report", perm_set)
self.assertNotIn(("can_csv_upload", "Database"), perm_set)
def assert_can_gamma(self, perm_set):
self.assert_can_read("Dataset", perm_set)
@ -1351,8 +1352,7 @@ class TestRolePermission(SupersetTestCase):
self.assert_can_all("CssTemplate", perm_set)
self.assert_can_all("Dataset", perm_set)
self.assert_can_read("Database", perm_set)
self.assertIn(("can_this_form_post", "CsvToDatabaseView"), perm_set)
self.assertIn(("can_this_form_get", "CsvToDatabaseView"), perm_set)
self.assertIn(("can_csv_upload", "Database"), perm_set)
self.assert_can_menu("Manage", perm_set)
self.assert_can_menu("Annotation Layers", perm_set)
self.assert_can_menu("CSS Templates", perm_set)

357
tests/unit_tests/databases/api_test.py
View File

@ -14,14 +14,11 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
# pylint: disable=unused-argument, import-outside-toplevel, line-too-long
import json
from datetime import datetime
from io import BytesIO
from typing import Any
from unittest.mock import Mock
from unittest.mock import ANY, Mock
from uuid import UUID
import pytest
@ -31,7 +28,11 @@ from pytest_mock import MockFixture
from sqlalchemy.orm.session import Session
from superset import db
from superset.commands.database.csv_import import CSVImportCommand
from superset.db_engine_specs.sqlite import SqliteEngineSpec
from tests.unit_tests.fixtures.common import create_csv_file
# pylint: disable=unused-argument, import-outside-toplevel, line-too-long
def test_filter_by_uuid(
@ -818,3 +819,351 @@ def test_oauth2_error(
}
]
}
@pytest.mark.parametrize(
"payload,cmd_called_with",
[
(
{
"file": (create_csv_file(), "out.csv"),
"table_name": "table1",
"delimiter": ",",
},
(
1,
"table1",
ANY,
{
"already_exists": "fail",
"delimiter": ",",
"file": ANY,
"table_name": "table1",
},
),
),
(
{
"file": (create_csv_file(), "out.csv"),
"table_name": "table2",
"delimiter": ";",
"already_exists": "replace",
"column_dates": "col1,col2",
},
(
1,
"table2",
ANY,
{
"already_exists": "replace",
"column_dates": ["col1", "col2"],
"delimiter": ";",
"file": ANY,
"table_name": "table2",
},
),
),
(
{
"file": (create_csv_file(), "out.csv"),
"table_name": "table2",
"delimiter": ";",
"already_exists": "replace",
"columns_read": "col1,col2",
"day_first": True,
"rows_to_read": "1",
"overwrite_duplicates": True,
"skip_blank_lines": True,
"skip_initial_space": True,
"skip_rows": "10",
"null_values": "None,N/A,''",
"column_data_types": '{"col1": "str"}',
},
(
1,
"table2",
ANY,
{
"already_exists": "replace",
"columns_read": ["col1", "col2"],
"null_values": ["None", "N/A", "''"],
"day_first": True,
"overwrite_duplicates": True,
"rows_to_read": 1,
"skip_blank_lines": True,
"skip_initial_space": True,
"skip_rows": 10,
"delimiter": ";",
"file": ANY,
"column_data_types": {"col1": "str"},
"table_name": "table2",
},
),
),
],
)
def test_csv_upload(
payload: dict[str, Any],
cmd_called_with: tuple[int, str, Any, dict[str, Any]],
mocker: MockFixture,
client: Any,
full_api_access: None,
) -> None:
"""
Test CSV Upload success.
"""
init_mock = mocker.patch.object(CSVImportCommand, "__init__")
init_mock.return_value = None
_ = mocker.patch.object(CSVImportCommand, "run")
response = client.post(
f"/api/v1/database/1/csv_upload/",
data=payload,
content_type="multipart/form-data",
)
assert response.status_code == 200
assert response.json == {"message": "OK"}
init_mock.assert_called_with(*cmd_called_with)
@pytest.mark.parametrize(
"payload,expected_response",
[
(
{
"file": (create_csv_file(), "out.csv"),
"delimiter": ",",
"already_exists": "fail",
},
{"message": {"table_name": ["Missing data for required field."]}},
),
(
{
"file": (create_csv_file(), "out.csv"),
"table_name": "",
"delimiter": ",",
"already_exists": "fail",
},
{"message": {"table_name": ["Length must be between 1 and 10000."]}},
),
(
{"table_name": "table1", "delimiter": ",", "already_exists": "fail"},
{"message": {"file": ["Field may not be null."]}},
),
(
{
"file": "xpto",
"table_name": "table1",
"delimiter": ",",
"already_exists": "fail",
},
{"message": {"file": ["Field may not be null."]}},
),
(
{
"file": (create_csv_file(), "out.csv"),
"table_name": "table1",
"delimiter": ",",
"already_exists": "xpto",
},
{"message": {"already_exists": ["Must be one of: fail, replace, append."]}},
),
(
{
"file": (create_csv_file(), "out.csv"),
"table_name": "table1",
"delimiter": ",",
"already_exists": "fail",
"day_first": "test1",
},
{"message": {"day_first": ["Not a valid boolean."]}},
),
(
{
"file": (create_csv_file(), "out.csv"),
"table_name": "table1",
"delimiter": ",",
"already_exists": "fail",
"header_row": "test1",
},
{"message": {"header_row": ["Not a valid integer."]}},
),
(
{
"file": (create_csv_file(), "out.csv"),
"table_name": "table1",
"delimiter": ",",
"already_exists": "fail",
"overwrite_duplicates": "test1",
},
{"message": {"overwrite_duplicates": ["Not a valid boolean."]}},
),
(
{
"file": (create_csv_file(), "out.csv"),
"table_name": "table1",
"delimiter": ",",
"already_exists": "fail",
"rows_to_read": 0,
},
{"message": {"rows_to_read": ["Must be greater than or equal to 1."]}},
),
(
{
"file": (create_csv_file(), "out.csv"),
"table_name": "table1",
"delimiter": ",",
"already_exists": "fail",
"skip_blank_lines": "test1",
},
{"message": {"skip_blank_lines": ["Not a valid boolean."]}},
),
(
{
"file": (create_csv_file(), "out.csv"),
"table_name": "table1",
"delimiter": ",",
"already_exists": "fail",
"skip_initial_space": "test1",
},
{"message": {"skip_initial_space": ["Not a valid boolean."]}},
),
(
{
"file": (create_csv_file(), "out.csv"),
"table_name": "table1",
"delimiter": ",",
"already_exists": "fail",
"skip_rows": "test1",
},
{"message": {"skip_rows": ["Not a valid integer."]}},
),
(
{
"file": (create_csv_file(), "out.csv"),
"table_name": "table1",
"delimiter": ",",
"already_exists": "fail",
"column_data_types": "{test:1}",
},
{"message": {"_schema": ["Invalid JSON format for column_data_types"]}},
),
],
)
def test_csv_upload_validation(
payload: Any,
expected_response: dict[str, str],
mocker: MockFixture,
client: Any,
full_api_access: None,
) -> None:
"""
Test CSV Upload validation fails.
"""
_ = mocker.patch.object(CSVImportCommand, "run")
response = client.post(
f"/api/v1/database/1/csv_upload/",
data=payload,
content_type="multipart/form-data",
)
assert response.status_code == 400
assert response.json == expected_response
def test_csv_upload_file_size_validation(
mocker: MockFixture,
client: Any,
full_api_access: None,
) -> None:
"""
Test CSV Upload validation fails.
"""
_ = mocker.patch.object(CSVImportCommand, "run")
current_app.config["CSV_UPLOAD_MAX_SIZE"] = 5
response = client.post(
f"/api/v1/database/1/csv_upload/",
data={
"file": (create_csv_file(), "out.csv"),
"table_name": "table1",
"delimiter": ",",
},
content_type="multipart/form-data",
)
assert response.status_code == 400
assert response.json == {
"message": {"file": ["File size exceeds the maximum allowed size."]}
}
current_app.config["CSV_UPLOAD_MAX_SIZE"] = None
@pytest.mark.parametrize(
"filename",
[
"out.xpto",
"out.exe",
"out",
"out csv",
"",
"out.csv.exe",
".csv",
"out.",
".",
"out csv a.exe",
],
)
def test_csv_upload_file_extension_invalid(
filename: str,
mocker: MockFixture,
client: Any,
full_api_access: None,
) -> None:
"""
Test CSV Upload validation fails.
"""
_ = mocker.patch.object(CSVImportCommand, "run")
response = client.post(
f"/api/v1/database/1/csv_upload/",
data={
"file": (create_csv_file(), filename),
"table_name": "table1",
"delimiter": ",",
},
content_type="multipart/form-data",
)
assert response.status_code == 400
assert response.json == {"message": {"file": ["File extension is not allowed."]}}
@pytest.mark.parametrize(
"filename",
[
"out.csv",
"out.txt",
"out.tsv",
"spaced name.csv",
"spaced name.txt",
"spaced name.tsv",
"out.exe.csv",
"out.csv.csv",
],
)
def test_csv_upload_file_extension_valid(
filename: str,
mocker: MockFixture,
client: Any,
full_api_access: None,
) -> None:
"""
Test CSV Upload validation fails.
"""
_ = mocker.patch.object(CSVImportCommand, "run")
response = client.post(
f"/api/v1/database/1/csv_upload/",
data={
"file": (create_csv_file(), filename),
"table_name": "table1",
"delimiter": ",",
},
content_type="multipart/form-data",
)
assert response.status_code == 200

22
tests/unit_tests/fixtures/common.py
View File

@ -14,8 +14,9 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
import csv
from datetime import datetime
from io import BytesIO, StringIO
import pytest
@ -23,3 +24,22 @@ import pytest
@pytest.fixture
def dttm() -> datetime:
return datetime.strptime("2019-01-02 03:04:05.678900", "%Y-%m-%d %H:%M:%S.%f")
def create_csv_file(data: list[list[str]] | None = None) -> BytesIO:
data = (
[
["Name", "Age", "City"],
["John", "30", "New York"],
]
if not data
else data
)
output = StringIO()
writer = csv.writer(output)
for row in data:
writer.writerow(row)
output.seek(0)
bytes_buffer = BytesIO(output.getvalue().encode("utf-8"))
return bytes_buffer

70
tests/unit_tests/forms_tests.py
View File

@ -1,70 +0,0 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
import contextlib
import tempfile
from typing import Optional
import pytest
from flask_wtf.file import FileField
from wtforms import Form, ValidationError
from superset.forms import FileSizeLimit
def _get_test_form(size_limit: Optional[int]) -> Form:
class TestForm(Form):
test = FileField("test", validators=[FileSizeLimit(size_limit)])
return TestForm()
@contextlib.contextmanager
def _tempfile(contents: bytes):
with tempfile.NamedTemporaryFile() as f:
f.write(contents)
f.flush()
yield f
def test_file_size_limit_pass() -> None:
"""Permit files which do not exceed the size limit"""
limit = 100
form = _get_test_form(limit)
with _tempfile(b"." * limit) as f:
form.test.data = f
assert form.validate() is True
def test_file_size_limit_fail() -> None:
"""Reject files which are too large"""
limit = 100
form = _get_test_form(limit)
with _tempfile(b"." * (limit + 1)) as f:
form.test.data = f
assert form.validate() is False
def test_file_size_limit_ignored_if_none() -> None:
"""Permit files when there is no limit"""
form = _get_test_form(None)
with _tempfile(b"." * 200) as f:
form.test.data = f
assert form.validate() is True