Skip to main content

WebSearch

WebSearch Node Documentation

Overview​

The WebSearch node enables your workflow to automatically search the internet and extract information from web pages. This powerful automation tool can gather data from search engines or scrape specific websites, making it perfect for market research, competitor analysis, lead generation, and content discovery workflows.

What This Node Does​

The WebSearch node acts as your workflow's research assistant, automatically:

  • Searching the web using Bing Search Engine API
  • Extracting specific information from web pages using custom URL scraping
  • Processing search results and formatting them for use in subsequent workflow steps
  • Handling multiple search queries and organizing results efficiently

Configuration Parameters​

Search Method Selection​

Field Name: mode

  • Type: Dropdown menu with options:
    • Custom URL with XPath parser: Scrapes data from specific web pages using XPath selectors to extract precise information
    • Bing Search Engine API: Performs web searches using Microsoft's Bing search engine and returns structured results
  • Default Value: Custom URL with XPath parser
  • Simple Description: Choose how you want to search and extract information from the web
  • When to Change This: Select "Custom URL" when you know exactly which websites to scrape, or "Bing Search Engine API" when you need to search the entire web for information
  • Business Impact: The right search method ensures you get accurate, relevant data while staying within your budget and compliance requirements

Search Method Options​

Custom URL with XPath Parser​

This method is perfect when you need to extract specific information from known websites, such as:

  • Product prices from competitor websites
  • News articles from specific publications
  • Contact information from company websites
  • Real estate listings from property sites

Key Benefits:

  • Precise data extraction from specific page elements
  • No API costs or rate limits
  • Works with any publicly accessible website
  • Highly customizable for different page structures

Bing Search Engine API​

This method is ideal when you need to search the entire web for information, such as:

  • Finding companies in specific industries
  • Researching trending topics or keywords
  • Discovering new competitors or market opportunities
  • Gathering comprehensive information on any topic

Key Benefits:

  • Access to billions of web pages through Bing's search index
  • Structured search results with titles, descriptions, and URLs
  • Advanced search filtering and customization options
  • Reliable, enterprise-grade search infrastructure

Real-World Use Cases​

Market Research Automation​

Business Situation: A marketing agency needs to monitor competitor pricing across multiple e-commerce sites daily to adjust their clients' pricing strategies.

What You'll Configure:

  • Select "Custom URL with XPath parser" from the search method dropdown
  • Enter competitor product page URLs
  • Configure XPath selectors to extract price information
  • Set up data formatting for easy analysis

What Happens: The workflow automatically visits competitor websites, extracts current pricing data, and formats it into a structured report that updates your pricing dashboard.

Business Value: Saves 20+ hours per week of manual price checking and ensures your pricing remains competitive in real-time.

Lead Generation Through Web Research​

Business Situation: A B2B sales team wants to automatically find and qualify potential customers by searching for companies that mention specific keywords or technologies on their websites.

What You'll Configure:

  • Choose "Bing Search Engine API" from the search method dropdown
  • Set up search queries targeting your ideal customer profile
  • Configure result filtering to focus on relevant company websites
  • Enable data extraction for contact information

What Happens: The workflow searches the web for companies matching your criteria, visits their websites to gather additional information, and creates a qualified lead list with contact details.

Business Value: Increases lead generation efficiency by 300% and improves lead quality through automated qualification processes.

Content Research and Monitoring​

Business Situation: A content marketing team needs to track mentions of their brand, monitor industry trends, and discover content opportunities across the web.

What You'll Configure:

  • Select "Bing Search Engine API" for comprehensive web coverage
  • Set up multiple search queries for brand mentions and industry keywords
  • Configure result processing to extract relevant content snippets
  • Enable automatic categorization of findings

What Happens: The workflow continuously monitors the web for relevant content, organizes findings by topic and relevance, and alerts your team to important mentions or trending topics.

Business Value: Reduces manual research time by 80% and ensures you never miss important industry developments or brand mentions.

Adding the Node​

  1. Drag the WebSearch node from the left panel onto your workflow canvas
  2. Connect it to the previous node using the arrow connector
  3. Ensure your previous node provides the search terms or URLs needed

Configuring Search Method​

  1. Click on the WebSearch node to open the settings panel
  2. In the "Type" dropdown, select your preferred search method:
    • Choose "Custom URL with XPath parser" for targeted website scraping
    • Choose "Bing Search Engine API" for comprehensive web searches
  3. The configuration options will automatically update based on your selection

Method-Specific Configuration​

The WebSearch node will display different configuration options depending on your selected search method. Each method has its own specialized settings panel that appears below the main type selection.

For Custom URL Method: You'll see options for URL configuration, XPath selectors, and data extraction settings.

For Bing API Method: You'll see options for API keys, search parameters, result filtering, and output formatting.

Integration with Other Nodes​

Input Requirements​

The WebSearch node works best when connected to nodes that provide:

  • Search terms or keywords (from form inputs, databases, or previous processing)
  • URLs to scrape (from spreadsheets, databases, or manual entry)
  • Dynamic search parameters (from user inputs or calculated values)

Output Capabilities​

The WebSearch node provides structured data that works perfectly with:

  • Data Processing Nodes: Clean and format extracted information
  • Database Nodes: Store search results for future use
  • Email Nodes: Send reports with research findings
  • Spreadsheet Nodes: Export data to Excel or Google Sheets
  • Decision Nodes: Route workflows based on search results

Industry Applications​

E-commerce and Retail​

Common Challenge: Manually tracking competitor prices, product availability, and market trends across dozens of websites.

How This Node Helps: Automatically monitors competitor websites, extracts pricing and product information, and alerts you to significant changes.

Configuration Recommendations:

  • Use "Custom URL with XPath parser" for known competitor sites
  • Set up multiple URL configurations for different product categories
  • Enable regular scheduling for continuous monitoring
  • Configure data validation to ensure accurate price extraction

Results: Retailers see 40% faster response to market changes and 25% improvement in competitive positioning.

Real Estate and Property Management​

Common Challenge: Gathering comprehensive market data from multiple listing services and property websites for investment analysis.

How This Node Helps: Automatically searches property listings, extracts key details like prices, locations, and features, and compiles comprehensive market reports.

Configuration Recommendations:

  • Use "Bing Search Engine API" for broad market searches
  • Configure location-based search parameters
  • Set up data extraction for property details and pricing
  • Enable result filtering by property type and price range

Results: Real estate professionals save 15+ hours per week on market research and make more informed investment decisions.

Digital Marketing and SEO​

Common Challenge: Monitoring brand mentions, tracking competitor content strategies, and identifying link-building opportunities across the web.

How This Node Helps: Continuously searches for brand mentions, analyzes competitor content, and discovers high-value websites for outreach campaigns.

Configuration Recommendations:

  • Use "Bing Search Engine API" for comprehensive web monitoring
  • Set up branded and competitor keyword searches
  • Configure result analysis for sentiment and relevance
  • Enable automatic categorization of findings

Results: Marketing teams increase their online visibility by 60% and reduce manual monitoring time by 75%.

Best Practices​

Search Method Selection​

  • Use Custom URL when you have specific websites to monitor and need precise data extraction
  • Use Bing API when you need comprehensive web coverage and don't know all relevant sources
  • Consider your budget: Custom URL has no API costs, while Bing API charges per search query

Data Quality and Reliability​

  • Always test your configurations with sample data before running full workflows
  • Set up data validation to catch extraction errors or format changes
  • Monitor your search results regularly to ensure continued accuracy
  • Keep backup configurations in case websites change their structure

Performance Optimization​

  • Limit search results to what you actually need to reduce processing time
  • Use specific search terms to get more relevant results
  • Schedule intensive searches during off-peak hours
  • Consider breaking large search tasks into smaller, parallel workflows

Compliance and Ethics​

  • Respect website terms of service and robots.txt files
  • Implement reasonable delays between requests to avoid overloading servers
  • Only extract publicly available information
  • Consider reaching out to website owners for permission when scraping large amounts of data

The WebSearch node transforms manual research tasks into automated, scalable processes that deliver consistent, high-quality results while freeing your team to focus on analysis and decision-making rather than data collection.