Spark SQL Planner
Optimizer优化后最终得到的还是Logical Plan，Planner会将Logical Plan转化为 Physical Plan lazy val sparkPlan: SparkPlan = { SparkSession.setActiveSession(sparkSession) // TODO: We use next(), i.e. take the first...
Spark SQL Optimizer
Optimizer与一样，也是继承自RuleExecutor,匹配规则对SQL进行优化 lazy val withCachedData: LogicalPlan = { assertAnalyzed() assertSupported() sparkSession.sharedState.cacheManager.useCachedData(analyzed) } lazy ...
Spark SQL Analyzer
Analyzer结合 catalog 进行绑定,生成 Resolved Logical Plan，在其父类RuleExecutor中匹配相应规则 lazy val analyzed: LogicalPlan = { SparkSession.setActiveSession(sparkSession) sparkSession.sessionState.analyzer.execu...
Spark SQL QueryExecution
1.sql 通过SqlParser 解析成 Unresolved Logical Plan; 2.analyzer 结合catalog 进行绑定,生成 Logical Plan; 3.optimizer 对 Logical Plan 优化,生成 Optimized LogicalPlan; 4.SparkPlan 将 Optimized LogicalPlan 转换成 Physical...
V2ray+WebSocket+TLS+Nginx
环境 centos7 安装certbot yum -y install certbot 生成key 注意需要开放端口443,80，由于vps没开放端口，导致没法生成key certbot certonly --standalone -d xxyy.tk -m xxyy@gmail.com 生成的文件位于 /etc/letsencrypt/live/xxyy.tk/privkey.pe...
StrucuredStreaming window
打印窗口示例 import org.apache.spark.sql.streaming.{OutputMode, ProcessingTime} import org.apache.spark.sql.{DataFrame, SparkSession} import org.apache.spark.sql.functions._ object StructuredNetworkWind...
Customize the Favicon
In Chirpy, the image files of Favicons are placed in assets/img/favicons/. You may need to replace them with your own. So let’s see how to customize these Favicons. Whit a square image (PNG, JPG o...
Getting Started
Preparation Follow the Jekyll Docs to complete the installtion of basic environment (Ruby, RubyGem, Bundler and Jekyll). In order to use the script tools to save time, we also need to install Pyth...
Writing a New Post
Naming and Path Create a new file name with the format YYYY-MM-DD-title.md then put it into _post of the root directory. Front Matter Basically, you need to fill the Front Matter as below at the...