#
## Step 1: Prepare Confluence for Data Export
1. Log in to your Confluence instance (https://shivaramb1702-1740378806876.atlassian.net)
2. Generate an API token:
- Click on your profile picture in the top-right corner
- Go to "Account Settings" > "Security" > "API tokens"
- Click "Create API token"
- Provide a label (e.g., "ADF Migration")
- Copy and securely store the generated token
## Step 2: Set Up Azure Resources
### Create a Resource Group
1. Sign in to the Azure Portal (https://portal.azure.com)
2. Click "Create a resource" in the upper left corner
3. Search for "Resource group" and select it
4. Click "Create"
5. Fill in the details:
- Subscription: Select your subscription
- Resource group name: `confluence-migration-rg`
- Region: Central India (or your preferred region)
6. Click "Review + create" then "Create"
### Create Azure Blob Storage Account
1. In the Azure Portal, navigate to your new resource group
2. Click "Add" or "Create resources"
3. Search for "Storage account" and select it
4. Click "Create"
5. Fill in the details:
- Subscription: Select your subscription
- Resource group: `confluence-migration-rg`
- Storage account name: Create a unique name (e.g., `confluencemigration12345`)
- Region: Central India (same as resource group)
- Performance: Standard
- Redundancy: Locally-redundant storage (LRS)
6. Click "Review" then "Create"
7. Once deployed, navigate to the storage account
8. Go to "Containers" in the left menu
9. Click "+ Container"
10. Create a container named "test" (or your preferred name)
11. Set public access level to "Private"
12. Click "Create"
### Create Azure Key Vault (Optional but Recommended)
1. In the Azure Portal, navigate to your resource group
2. Click "Add" or "Create resources"
3. Search for "Key Vault" and select it
4. Click "Create"
5. Fill in the details:
- Subscription: Select your subscription
- Resource group: `confluence-migration-rg`
- Key vault name: Create a unique name (e.g., `confluence-kv-12345`)
- Region: Central India (same as resource group)
- Pricing tier: Standard
6. Click "Review + create" then "Create"
7. Once deployed, go to "Secrets" in the left menu
8. Click "+ Generate/Import"
9. Add your Confluence API token:
- Name: `confluence-api-token`
- Value: Paste your Confluence API token
10. Click "Create"
## Step 3: Create Azure Data Factory
1. In the Azure Portal, navigate to your resource group
2. Click "Add" or "Create resources"
3. Search for "Data Factory" and select it
4. Click "Create"
5. Fill in the details:
- Subscription: Select your subscription
- Resource group: `confluence-migration-rg`
- Name: `confluence-migration-df`
- Version: V2
- Region: Central India (same as resource group)
- Git configuration: Configure Git later (uncheck)
6. Click "Review + create" then "Create"
7. Once deployment is complete, click "Go to resource"
8. Click "Launch Studio" to open Azure Data Factory Studio
## Step 4: Configure Linked Services
### Create HTTP Linked Service for Confluence API
1. In ADF Studio, click on "Manage" in the left sidebar
2. Click "Linked services" then "+ New"
3. Search for "HTTP" and select it
4. Configure the HTTP linked service:
- Name: `HttpServer1`
- Base URL: `https://shivaramb1702-1740378806876.atlassian.net/wiki/rest/api/` (include the trailing slash)
- Authentication type: "Basic"
- Username: `shivaramb1702@gmail.com`
- Password: Enter your Confluence API token
- Test connection: Click to verify it works
5. Click "Create"
### Create REST Linked Service for Confluence API
1. In the "Linked services" section, click "+ New"
2. Search for "REST" and select it
3. Configure the REST linked service:
- Name: `ConfluenceRESTAPI`
- Base URL: `https://shivaramb1702-1740378806876.atlassian.net/wiki/rest/api/` (include the trailing slash)
- Authentication type: "Basic"
- Username: `shivaramb1702@gmail.com`
- Password: Enter your Confluence API token
- Test connection: Click to verify it works
4. Click "Create"
### Create Azure Blob Storage Linked Service
1. In the "Linked services" section, click "+ New"
2. Search for "Azure Blob Storage" and select it
3. Configure:
- Name: `AzureBlobStorage1`
- Authentication method: Account key
- Account selection method: From Azure subscription
- Azure subscription: Select your subscription
- Storage account name: Select the storage account you created earlier
- Test connection: Click to verify it works
4. Click "Create"
## Step 5: Create Datasets
### Create HTTP Dataset for Confluence Spaces
1. In ADF Studio, click the "Author" tab in the left sidebar
2. Click "+" and select "Dataset"
3. Search for "HTTP" and select it
4. Configure:
- Name: `ConfluenceSpacesDataset`
- Linked service: Select `HttpServer1`
- Relative URL: `space?limit=100`
- Request method: GET
- Response format: JSON
5. Click "OK"
### Create REST Dataset for Confluence Pages
1. Click "+" and select "Dataset"
2. Search for "REST" and select it
3. Configure:
- Name: `RestResource1`
- Linked service: Select `ConfluenceRESTAPI`
- Relative URL: `space/@{pipeline().parameters.spaceKey}/content?type=page&expand=body.storage,version,ancestors,children.attachment,children.comment&limit=100`
- Note: This will use pipeline parameters that we'll define later
4. Click "OK"
### Create Azure Blob Storage Dataset for Pages
1. Click "+" and select "Dataset"
2. Search for "Azure Blob Storage" and select it
3. Select format as "JSON"
4. Configure:
- Name: `Json1`
- Linked service: Select `AzureBlobStorage1`
- File path: Container: `test`, Directory: `@{pipeline().parameters.spaceName}/pages`
- File name: `pages.json`
- Import schema: None
5. Click "OK"
### Create Generic Azure Blob Storage Dataset
1. Click "+" and select "Dataset"
2. Search for "Azure Blob Storage" and select it
3. Select format as "JSON"
4. Configure:
- Name: `AzureBlobStorageDataset`
- Linked service: Select `AzureBlobStorage1`
- Import schema: None
5. Click "OK"
## Step 6: Create Pipelines
### Pipeline 1: ExtractPagesAndBlogs
1. In ADF Studio's "Author" tab, click "+" and select "Pipeline"
2. Name the pipeline: `ExtractPagesAndBlogs`
3. Add parameters to the pipeline:
- Click on "Parameters" at the bottom of the screen
- Add parameter "spaceKey" (type: String)
- Add parameter "spaceName" (type: String)
4. Add a Copy Data activity:
- Name: `Data activity for Pages`
- Source:
- Source dataset: `RestResource1`
- Use query: No
- Sink:
- Sink dataset: `Json1`
- Copy behavior: None
5. Add another Copy Data activity:
- Name: `Copy Data activity for Blogs`
- Connect it to run after "Data activity for Pages" (use the success dependency)
- Source:
- Source dataset: Create a new REST dataset similar to RestResource1 but with:
- Relative URL: `space/@{pipeline().parameters.spaceKey}/content?type=blogpost&expand=body.storage,version&limit=100`
- Sink:
- Sink dataset: Create a new JSON dataset similar to Json1 but with:
- File path: Container: `test`, Directory: `@{pipeline().parameters.spaceName}/blogs`
- File name: `blogs.json`
6. Add a Web activity:
- Name: `GetAttachments`
- Connect it to run after "Copy Data activity for Blogs" (use the success dependency)
- URL: `https://shivaramb1702-1740378806876.atlassian.net/wiki/rest/api/space/@{pipeline().parameters.spaceKey}/content?type=attachment&expand=version&limit=100`
- Method: GET
- Authentication: Basic
- Username: `shivaramb1702@gmail.com`
- Password: Your Confluence API token
### Pipeline 2: GetConfluenceSpaces
1. Create a new pipeline: `GetConfluenceSpaces`
2. Add a Web activity:
- Name: `GetSpaces`
- URL: `https://shivaramb1702-1740378806876.atlassian.net/wiki/rest/api/space?limit=100`
- Method: GET
- Authentication: Basic
- Username: `shivaramb1702@gmail.com`
- Password: Your Confluence API token (do not hardcode - use a parameter or Key Vault)
3. Add a ForEach activity:
- Name: `ForEachSpace`
- Connect it to run after "GetSpaces" (use the success dependency)
- Items: `@activity('GetSpaces').output.results`
- Sequential: Unchecked (to allow parallel processing)
4. Inside the ForEach activity, add an Execute Pipeline activity:
- Name: `Execute Pipeline1`
- Invoked pipeline: `ExtractPagesAndBlogs`
- Parameters:
- spaceKey: `@{item().key}`
- spaceName: `@{item().name}`
- Wait on completion: Checked
### Pipeline 3: MigrateAllConfluenceContent
1. Create a new pipeline: `MigrateAllConfluenceContent`
2. Add an Execute Pipeline activity:
- Name: `Execute Pipeline1`
- Invoked pipeline: `GetConfluenceSpaces`
- Wait on completion: Checked
## Step 7: Validate and Debug
1. In ADF Studio, navigate to your `MigrateAllConfluenceContent` pipeline
2. Click "Debug" to test the pipeline
3. Monitor the pipeline execution in the "Output" tab
4. Check for any errors and fix them
5. Verify that data is being correctly written to your Azure Blob Storage
## Step 8: Deploy and Schedule
1. Once your pipeline is working correctly, click "Publish All" to save your changes
2. To schedule the pipeline to run periodically:
- Go to the "Manage" tab
- Click "Triggers" then "+ New"
- Configure a schedule trigger with your desired frequency
- Associate it with the `MigrateAllConfluenceContent` pipeline
## Common Issues to Watch For
1. **API Token Security**: Never hardcode your API token in the pipeline like in the template example. Always use parameters or Key Vault.
2. **URL Format**: Ensure your base URLs end with a trailing slash (`/`) to avoid URL concatenation issues.
3. **Parameter Passing**: Make sure parameters are correctly passed between pipelines, especially in the ForEach activity.
4. **Container Name**: The template hardcodes "test" as the container name. Change this to match your container name if different.
5. **Authentication**: Verify your authentication credentials work by testing connections for all linked services.
6. **Pagination**: For large Confluence spaces, you may need to implement pagination to retrieve all content.
7. **Error Handling**: Add error handling activities to make your pipeline more robust.
By following these detailed steps, you should be able to successfully create an Azure Data Factory solution to migrate your Confluence content to Azure Blob Storage.
Citations:
[1] https://ppl-ai-file-upload.s3.amazonaws.com/web/direct-files/31195055/38a584fc-94f4-48ec-ba9f-0e6d4f22ceb5/ARMTemplateForFactory.json
[2] https://ppl-ai-file-upload.s3.amazonaws.com/web/direct-files/31195055/a593d540-822b-44a3-9009-2dc079436db2/ARMTemplateParametersForFactory.json
[3] https://ppl-ai-file-upload.s3.amazonaws.com/web/direct-files/31195055/308b731a-03ed-474d-8aa1-331fd7125aa8/confluence-migration-df_ARMTemplateForFactory.json
[4] https://ppl-ai-file-upload.s3.amazonaws.com/web/direct-files/31195055/4bf1a524-d3aa-4466-9017-509b74b506cf/confluence-migration-df_ARMTemplateParametersForFactory.json
[5] https://ppl-ai-file-upload.s3.amazonaws.com/web/direct-files/31195055/cb546b96-daab-4016-b567-e2eb5b3a8958/ArmTemplate_0.json
[6] https://ppl-ai-file-upload.s3.amazonaws.com/web/direct-files/31195055/d02eb408-7f37-474c-bb5b-54b0564f28d5/ArmTemplate_master.json
[7] https://ppl-ai-file-upload.s3.amazonaws.com/web/direct-files/31195055/e9e276e4-788b-4d60-a3fc-e22b64761802/ArmTemplateParameters_master.json
---
Answer from Perplexity: pplx.ai/share
Comments
Post a Comment